Devices, Systems, and Methods that Observe and Classify Real-World Activity Relating to an Observed Object, and Track and Disseminate State Relating the Observed Object

ABSTRACT

An intelligent monitoring device including a processor and an accelerometer and/or a device that includes at least one processor and at least one image sensor. The intelligent monitoring device is configured to observe at least one machine. The intelligent monitoring device is further configured to utilize its processor, or the processor of a coupled system, to recognize actions carried out on or by the at least one machine and infer the state of the machine. The intelligent monitoring device or the coupled system is further configured to provide alerts or help respond to queries about the status of the at least one machine. For example, the intelligent monitoring camera will infer the state of a washing machine based on its observations and provide an alert (optionally only to the user recognized to have loaded the washer) when the washing machine is ready to be unloaded.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.16/777,860, filed Jan. 30, 2020, which claims the benefit of U.S.Provisional Application 62/799,004, filed Jan. 30, 2019. All the abovereferenced Provisional and Non-Provisional applications and anyresulting Issued patents are herein incorporated by reference in theirentirety.

COPYRIGHTED MATERIAL

A portion of the disclosure of this patent document contains materialwhich is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by anyone of the patent documentor the patent disclosure, as it appears in the Patent and TrademarkOffice patent file or records, but otherwise reserves all copyrightrights whatsoever.

REFERENCE TO A “SEQUENCE LISTING,” A TABLE, OR A COMPUTER PROGRAMLISTING APPENDIX SUBMITTED AS AN ASCII TEXT FILE

The official copy of the computer program listing appendix is submitted,with information previously provided on pages 87-195 of thespecification as originally filed in the Parent U.S. Non-Provisionalapplication Ser. No. 16/777,860, and/or the U.S. Provisional Application62/799,004, as an ASCII formatted text file via EFS-Web, with a filename of “OBSERVER_COMPUTER_PROGRAM_APPENDIX.txt”, a creation date ofNov. 19, 2021, and a size of 262183 bytes. The computer program listingfiled via EFS-Web is part of the specification, is incorporated in itsentirety by reference herein, and contains no new mater which goesbeyond the disclosure of the application as filed.

BACKGROUND 1. Field of the Invention

The invention generally relates to observer devices and systems that areable to recognize appliances and other machines, people, or objectsbased on sensor data such as accelerometer data, visual imaging, depthsensor data, thermal image sensor data, sound data, and any other datatype. The observer devices that are the subject of this invention mayfurther be trained or configured to recognize actions performed on or bythe appliances, machines, people and objects that the devices areobserving. The observer camera devices may infer a state or condition ofthe appliance, machine, or object based on its observations of theobserved device and other inputs.

Many people and businesses have legacy appliances/machines, or newappliances/machines that may or may not able to connect to the internet.Newer appliances that do connect to the internet are either expensive ormay present a security risk that consumers do not want to take. Forexample, some appliances such as ovens, washers, dryers and dishwashersmay be started or controlled over the Internet. If a malicious actortakes control of these devices, it may cause damage or danger for theappliance owner. Thus, many people do not want their appliances toconnect to the internet to be susceptible to attack in this manner.

However, it can be beneficial to know the state of machines orappliances utilizing the Internet. Knowing the state of each appliancecan be particularly challenging and useful in a multi-person householdwhere one person may use a machine without informing the others.Furthermore, being able to receive notifications, reminders, orresponses to queries about the state of a device can ensure that usersunload appliances in a timely manner, do not waste time preparing toload a device that is in use by another member, and utilize householdappliances efficiently.

Accordingly, there is a need for a device that will allows legacy,non-Internet connected devices, or newer Internet enabled devices tocommunicate their state such that appropriate people can easily obtainthe information on their phone or through a voice assistant in a securemanner. Furthermore, there is a need for a device to communicate suchinformation in such a way that does not potentially provide the abilityfor malicious third-parties to control machines or appliances.

Still further, it is beneficial to track the state of other objects andpeople around the house or workplace in order to ensure everything is OKor track efficiency. Thus, there is a need for a device to observepeople and employees and notify appropriate people of issues that arise.In some forms, privacy can be maintained by only sending textualdescriptions of the state of a person when an abnormal state is detectedor providing pictures if authorized. Traditional programming techniquesmay be used to determine and monitor the state of appliances and othermachines. Artificial Intelligent systems, such as deep neural networks,may also be used and implemented with various processors, sensors,cameras and other devices to observe and determine the states ofmonitored machines.

Traditional software programming uses Boolean-based logic that can betested to confirm that the software does what it was designed to do,using tools and methodologies established over the last few decades.

It can be beneficial to use deep neural network machine learning models.However, it is hard to debug neural network algorithms because they area black box once trained. Machine learning is a black box programmingmethod in which computers program themselves with data, producingprobabilistic logic that diverges from the true-and-false tests used toverify systems programmed with traditional Boolean logic methods.

There is a need to facilitate debugging deep neural networks and thesystems built upon them. Currently, the conventional wisdom is to createa custom machine learning model that is trained to perform all aspectsof the logic on the device, including state logic. In many cases this isinefficient as it requires each device to be re-trained any time thereis a change. The training process can be complicated and computationallyexpensive.

There is benefit to defining logic of state machine without the use ofmachine learning training methods, and then apply widely applicablemachine learning models to define the states or transitions. This ismuch more efficient, as computing resources (e.g., power, carbon, etc.)do not need to be expended to come up with the ability to replicate thestate machine and define states and transitions in that context. Thecomputing resources can be utilized to create and refine widelyapplicable machine learning classifiers that may be used to definestates and transitions based on pictures (e.g., ImageNet or extensionsthereof, etc.), video (e.g., UCF-101 or extensions thereof, etc.),audio, point clouds, or other data that is sensed by an observer device.To the extent these machine learning models need to be adapted,retrained or replaced, they can be swapped out while leaving the otherclassifiers and state machine logic untouched. The devices that utilizemachine learning models can host many different neural networks that areeither run in parallel or swapped out depending on the state of thedevice. This will facilitate creation of devices that implement machinelearning algorithms, and improve the functioning, testability,predictability, and maintenance of “artificially intelligent” devices.

It can be useful to monitor machines via the Internet or other wired andwireless connections. However, in many cases it is not practical toupdate large home appliances or industrial machines; indeed, it can bewasteful to dispose of a large appliance because it simply does not havethe newest connectivity accessory available. There is a need to connectlegacy home appliances without modifying the appliances.

2. Description of Related Materials

To reduce the complexity and length of the Detailed Specification, andto fully establish the state of the art in certain areas of technology,Applicant herein expressly incorporates by reference all the followingmaterials identified in each numbered paragraph below. The Cite Numbersreferred to herein are set forth in the Information Disclosure Statement(IDS) filed with this application and incorporated by reference herein.

The related art shows the novel and non-obvious nature of the presentinvention including secondary considerations of non-obviousness such asthe long-felt need and failure of others to achieve the presentinvention. All referenced materials are herein incorporated by referencein its entirety.

2.C Other Technologies

Applicant believes that the material incorporated above is“non-essential” in accordance with 37 CFR 1.57, because it is referredto for purposes of indicating the background of the invention orillustrating the state of the art. However, if the Examiner believesthat any of the above-incorporated material constitutes “essentialmaterial” within the meaning of 37 CFR 1.57(c)(1)-(3), Applicant willamend the specification to expressly recite the essential material thatis incorporated by reference as allowed by the applicable rules.

SUMMARY OF THE INVENTION

Aspects and applications of the invention presented here are describedbelow in the drawings and detailed description of the invention. Unlessspecifically noted, it is intended that the words and phrases in thespecification and the claims be given their plain, ordinary, andaccustomed meaning to those of ordinary skill in the applicable arts.The inventor is fully aware that he can be his own lexicographer ifdesired. The inventor expressly elects, as his own lexicographers, touse only the plain and ordinary meaning of terms in the specificationand claims unless he clearly states otherwise and then further,expressly sets forth the “special” definition of that term and explainshow it differs from the plain and ordinary meaning. Absent such clearstatements of intent to apply a “special” definition, it is theinventor's intent and desire that the simple, plain and ordinary meaningto the terms be applied to the interpretation of the specification andclaims.

The inventor is also aware of the normal precepts of English grammar.Thus, if a noun, term, or phrase is intended to be furthercharacterized, specified, or narrowed in some way, then such noun, term,or phrase will expressly include additional adjectives, descriptiveterms, or other modifiers in accordance with the normal precepts ofEnglish grammar. Absent the use of such adjectives, descriptive terms,or modifiers, it is the intent that such nouns, terms, or phrases begiven their plain, and ordinary English meaning to those skilled in theapplicable arts as set forth above.

In that regard, the use of the word “coupled” or “connected” impliesthat the elements may be directly connected or may be indirectlyconnected or coupled through one or more intervening elements unless itis specifically noted that there must be a direct connection.

Further, the inventor is fully informed of the standards and applicationof the special provisions of 35 U.S.C. § 112(f). Thus, the use of thewords “function,” “means” or “step” in the Detailed Description orDescription of the Drawings or claims is not intended to somehowindicate a desire to invoke the special provisions of 35 U.S.C. §112(f), to define the invention. To the contrary, if the provisions of35 U.S.C. § 112(f) are sought to be invoked to define the inventions,the claims will specifically and expressly state the exact phrases“means for” or “step for, and will also recite the word “function”(i.e., will state “means for performing the function of [insertfunction]”), without also reciting in such phrases any structure,material or act in support of the function. Thus, even when the claimsrecite a “means for performing the function of . . . “or “step forperforming the function of . . . ,” if the claims also recite anystructure, material or acts in support of that means or step, or thatperform the recited function, then it is the clear intention of theinventor not to invoke the provisions of 35 U.S.C. § 112(f). Moreover,even if the provisions of 35 U.S.C. § 112(f) are invoked to define theclaimed inventions, it is intended that the inventions not be limitedonly to the specific structure, material or acts that are described inthe preferred embodiments, but in addition, include any and allstructures, materials or acts that perform the claimed function asdescribed in alternative embodiments or forms of the invention, or thatare well known present or later-developed, equivalent structures,material or acts for performing the claimed function.

Headings, sections, and other similar designations are provided for theconvenience of the reader, and should not be used to limit, divide, orpartition the teachings of the variously claimed aspects of theinventions.

The aspects, features, and advantages will be apparent to those artisansof ordinary skill in the art from the DETAILED DESCRIPTION and DRAWINGS,and from the CLAIMS. However, without attempting to characterize orlimit the scope of inventions as they are described and claimed, some ofthe advantages of the various inventions are summarized below.

It is an object of the invention to track the state of household andindustrial appliances, machines, people, and objects with relativelyinexpensive “smart” or “Internet of Things” (IoT) devices that are ableto easily attach to or view the legacy machine in order to observe andinfer the state of the legacy machines.

It is yet another (and optionally independent) object of the inventionto implement use sensor-based devices to infer and track the state ofappliances, machines, people and objects of interest to a user.

It is yet another (and optionally independent) object of the inventionto use accelerometer sensor devices to infer and track the state ofappliances, machines, people and objects of interest to a user.

It is yet another (and optionally independent) object of the inventionto use heat gyroscope sensor devices to infer and track the state ofappliances, machines, people and objects of interest to a user.

It is yet another (and optionally independent) object of the inventionto use magnetometer sensor devices to infer and track the state ofappliances, machines, people and objects of interest to a user.

It is yet another (and optionally independent) object of the inventionto use heat inertial measurement unit devices to infer and track thestate of appliances, machines, people and objects of interest to a user.

It is yet another (and optionally independent) object of the inventionto use vibration sensor devices to infer and track the state ofappliances, machines, people and objects of interest to a user.

It is yet another (and optionally independent) object of the inventionto use security camera devices to infer and track the state ofappliances, machines, people and objects of interest to a user.

It is yet another (and optionally independent) object of the inventionto use heat sensor devices to infer and track the state of appliances,machines, people and objects of interest to a user.

It is yet another (and optionally independent) object of the inventionto use depth sensor devices to infer and track the state of appliances,machines, people and objects of interest to a user.

It is yet another (and optionally independent) object of the inventionto use sound sensor devices to infer and track the state of appliances,machines, people and objects of interest to a user.

It is yet another (and optionally independent) object of the inventionto use power usage sensor devices to infer and track the state ofappliances, machines, people and objects of interest to a user.

It is yet another (and optionally independent) object of the inventionto use multiple-sensor devices to infer and track the state ofappliances, machines, people and objects of interest to a user.

It is yet another (and optionally independent) object of the inventionto use any type of sensor device or devices with to infer and track thestate of appliances, machines, people and objects of interest to a userby correlating observed data to real-world actions performed on or bythe machine or sensor devices.

It is yet another (and optionally independent) object of the inventionto use machine learning techniques to correlating observed data toreal-world actions performed on or by the machine or sensor devices.

It is yet another (and optionally independent) object of the inventionto use neural networks to correlate observed data to real-world actionsperformed on or by the machine or sensor devices.

It is yet another (and optionally independent) object of the inventionto use convolutional neural networks to correlate observed data toreal-world actions performed on or by the machine or sensor devices.

It is yet another (and optionally independent) object of the inventionto use recurrent neural networks to correlate observed data toreal-world actions performed on or by the machine or sensor devices.

It is yet another (and optionally independent) object of the inventionto use long-term recurrent convolutional networks for visual recognitionand to correlate observed data to real-world actions performed on or bythe machine or sensor devices.

It is yet another (and optionally independent) object of the inventionto use long short-term memory neural networks to correlate observed datato real-world actions performed on or by the machine or sensor devices.

It is yet another (and optionally independent) object of the inventionto use deep learning to correlate observed data to real-world actionsperformed on or by the machine or sensor devices.

It is yet another (and optionally independent) object of the inventionto use a combination of neural networks and/or deep learning tocorrelate observed data to real-world actions performed on or by themachine or sensor devices.

It is yet another (and optionally independent) object of the inventionto use algorithmic programming to correlate observed data to real-worldactions performed on or by the machine or sensor devices.

It is yet another (and optionally independent) object of the inventionto use heuristic programming to correlate observed data to real-worldactions performed on or by the machine or sensor devices.

It is yet another (and optionally independent) object of the inventionto use any other methods to correlate observed data to real-worldactions performed on or by the machine or sensor devices.

It is yet another (and optionally independent) object of the inventionto track the state of a machine, appliance, object or person using afinite-state machine model.

It is yet another (and optionally independent) object of the inventionto track the state of a machine, appliance, object or person using atwo-state finite-state machine model.

It is yet another (and optionally independent) object of the inventionto track the state of a machine, appliance, object or person using athree-state finite-state machine model.

It is yet another (and optionally independent) object of the inventionto track the state of a machine, appliance, object or person using anabnormal state detector finite-state machine model.

It is yet another (and optionally independent) object of the inventionto track the state of a machine, appliance, object or person using anneglectable machine finite-state machine model.

It is yet another (and optionally independent) object of the inventionto track the state of a machine, appliance, object or person using anloadable machine finite-state machine model.

It is yet another (and optionally independent) object of the inventionto track the state of a machine, appliance, object or person using ansingle-user machine finite-state machine model.

It is yet another (and optionally independent) object of the inventionto determine the identity of users that act on machines.

It is yet another (and optionally independent) object of the inventionto Determine that a user has loaded a machine,

It is yet another (and optionally independent) object of the inventionto determine that the machine has run,

It is yet another (and optionally independent) object of the inventionto determine that the machine subsequently is done and ready for furtherprocessing (e.g., a user to unload the machine).

It is yet another (and optionally independent) object of the inventionto utilize messaging to communicate to a user that loaded a machine thatthe machine is ready to be unloaded.

It is yet another (and optionally independent) object of the inventionto utilize messaging to communicate to the user that was determined tohave last interacted with a machine (e.g., loaded the machine) that themachine needs further attention (e.g., it is done and is ready to beunloaded).

It is yet another (and optionally independent) object of the inventionto provide a backend state tracking database for appliances, machines,people (i.e., any object) so that other devices may query the backend todetermine the current or historical states of the tracked objects.

It is yet another (and optionally independent) object of the inventionto act on machines with actuators (e.g., trigger an infraredtransmitter, power relay, motion device, etc.).

It is yet another (and optionally independent) object of the inventionto alert users that devices have been left in a vulnerable or open state(e.g., “the garage is open!” “the oven is on!”).

It is yet another (and optionally independent) object of the inventionto utilize voice assistants (e.g., Google Home, Amazon Alexa, AppleSiri) to alert users of a change in the state of an observed object(e.g., “the dishwasher is done.”).

It is yet another (and optionally independent) object of the inventionto utilize voice assistants in the home to respond to queries (e.g., “isthe dishwasher done?”) about observed machines (e.g., “yes, thedishwasher is done.”).

It is yet another (and optionally independent) object of the inventionto help treat involuntary movement disorders such as trichotillomania(hair pulling) and other body-focused repetitive behavior classified asan impulse control disorders.

It is yet another (and optionally independent) object of the inventionto discipline dogs that are doing things they should not be (e.g.,“Chubs, stop!”) based on image recognition technology.

At least one of the above listed, unlisted, and other objects of theinvention may be achieved by an observer device with a processor, atriple-axis accelerometer, and a wireless communication interface (e.g.,a Bluetooth, Wi-Fi, ISM, LoRA, Cellular, etc.) wherein the processor isconfigured to: periodically record a series of measurements from atleast one axis of the triple-axis accelerometer over a period of time,process those measurements using a signal processing technique such as aFourier transform, compute the average value of the Fourier transformresult, compare the average value to a threshold value, determine, basedon the comparison, whether the device should cause a transition in astate machine model of an observed machine, and if the determination isyes, cause the state machine transition to occur.

At least one of the above listed, unlisted, and other objects of theinvention may be achieved by an observer device with a processor, animaging sensor, and a wireless communication interface (e.g., aBluetooth, Wi-Fi, ISM, LoRA, Cellular, etc.) wherein the processor isconfigured to: periodically record an image from the imaging sensor,pass the image through a deep neural network to obtain a result,determine, based on the result, whether the device should cause atransition in a state machine model of an observed machine, and if thedetermination is yes, cause the state machine transition to occur.

The above listed objects, and summary of how the objects might beachieved, are intended to assist with the understanding of theinvention(s), and shall not be used to limit the scope of claims as donein Pacing Technologies, LLC v. Garmin International, Inc., No. 14-1396(Fed. Cir. Feb. 18, 2015).

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

A more complete understanding of the present invention may be derived byreferring to the detailed description when considered in connection withthe following illustrative figures. In the figures, like referencenumbers refer to like elements or acts throughout the figures.

FIG. 1 depicts one example observer device.

FIG. 2A depicts another example observer device.

FIG. 2B depicts a schematic diagram of an example observer device.

FIG. 2C depicts yet another example observer device.

FIG. 2D depicts one physical form of an example observer device.

FIG. 2E depicts another physical form of an example observer device.

FIG. 2F depicts a physical form of an example image-based observerdevice.

FIG. 2G depicts a physical form of a sensor- or accelerometer-basedobserver device.

FIG. 2H depicts a physical form of a circuit board for a sensor- oraccelerometer-based observer device.

FIG. 2I depicts another view of a physical form of a circuit board for asensor- or accelerometer-based observer device.

FIG. 2J depicts a physical form of the circuit board for a sensor- oraccelerometer-based observer device in the case depicted in FIG. 2G.

FIG. 2K depicts a rear view of FIG. 2J.

FIG. 2L depicts a physical form of a sensor- or accelerometer-basedobserver device with a display screen indicating that a dishwasher isclean or done.

FIG. 2M depicts a physical form of a sensor- or accelerometer-basedobserver device with a display screen indicating that a dishwasher isrunning.

FIG. 2N depicts a physical form of a sensor- or accelerometer-basedobserver device with a display screen indicating that a dishwasher isdirty or ready to load.

FIG. 3A depicts one example system utilizing an observer device.

FIG. 3B depicts another example system utilizing an observer device.

FIG. 4 depicts one example method of determining the orientation of anobserver device relative to the ground using accelerometer data.

FIG. 5A depicts example accelerometer, temperature, andgyroscope/magnetometer data collected by an observer device attached toan example appliance.

FIG. 5B depicts experimental data collected from an experimental sensor-or accelerometer-based observer device.

FIG. 5C depicts a graph of the data presented in FIG. 5B.

FIG. 6A depicts utilizing a method of waveform analysis to determine anappliance state (e.g., running or not running, etc.) based on data suchas depicted in FIG. 5.

FIG. 6B depicts utilizing a neural network or machine learning model todetermine an appliance state (e.g., running or not running, etc.) basedon data such as depicted in FIG. 5.

FIG. 7A depicts an example accelerometer-based observer attached to anexample dishwasher appliance.

FIG. 7B depicts an example accelerometer-based observer attached to anexample top-load washing machine appliance.

FIG. 7C depicts an illustration of an example accelerometer-basedobserver attached to an example front-load washing machine or a dryerappliance.

FIG. 7D depicts an example accelerometer-based observer attached to anexample garage door.

FIG. 7E depicts an example observer with object or range detector, agyroscope, thermometer, and/or magnetometer, attached to an examplerefrigerator door.

FIG. 7F depicts an example observer with thermometer attached to anexample refrigerator door.

FIG. 8A depicts a two-state state diagram.

FIG. 8B depicts a three-state state diagram.

FIG. 8C depicts a neglectable machine state diagram.

FIG. 8D depicts a state diagram for a loadable machine.

FIG. 8E depicts a state diagram for determining the orientation of anobserver device.

FIG. 9A depicts an abnormal state detector (or generic machine) statediagram.

FIG. 9B depicts an openable machine (or neglectable machine) statediagram.

FIG. 9C depicts a neglectable machine (or openable machine) statediagram.

FIG. 9D depicts a single-user machine state diagram.

FIG. 9E depicts a loadable machine state diagram.

FIG. 9F depicts a cyclic machine state diagram.

FIG. 9G depicts an exemplary nested state machine diagram for anobserved dishwasher.

FIG. 9H depicts an exemplary parallel state machine diagram for anobserved refrigerator.

FIG. 10 depicts one possible example of a Neural Network (e.g., CNN LSTMMulti-Path, etc.) that may be used in various forms of the invention.

FIG. 11 depicts one possible process used in various forms of theinvention for obtaining training data, training the neural network,operating the neural network and refining the neural network over time.

FIG. 12A depicts a flowchart for causing notifications or announcementsto be transmitted to mobile devices or voice assistants based on anobservation from an observer device.

FIG. 12B depicts a main screen of one example mobile application.

FIG. 12C depicts an add new device screen of one example mobileapplication.

FIG. 12D depicts a detail view of a second example mobile applicationshowing that a dishwasher is running.

FIG. 12E depicts another detail view of the second example mobileapplication showing that a clothes dryer is done.

FIG. 12F depicts a main screen of a third example mobile application.

FIG. 12G depicts an add device screen of the third example mobileapplication.

FIG. 12H depicts a report management screen of the third example mobileapplication.

FIG. 12I depicts a device detail settings screen of the third examplemobile application.

FIG. 12J depicts a device detail display screen of the third examplemobile application.

FIG. 13A depicts a flowchart for updating an observer backend based onan observation from an observer device.

FIG. 13B depicts a flowchart for handling a voice assistant queryrelating to an observer and obtaining a vocalized response.

FIG. 13C depicts an intent listing associated with a voice assistantagent.

FIG. 13D depicts details for a “get appliance state” voice assistantagent intent.

FIG. 13E depicts an entities listing associated with the voice assistantagent.

FIG. 13F depicts details for the appliance entity associated with thevoice assistant agent.

FIG. 13G depicts details for the state entity associated with the voiceassistant agent.

FIG. 13H depicts the fulfillment options associated with the voiceassistant agent

FIG. 13I depicts an example conversation with the voice assistant agent.

FIG. 14 depicts an exemplary flowchart for the creation, training,running, and refining of a machine learning or AI state machine model.

FIG. 15A depicts an exemplary Graphic User Interface (GUI) for creatinga new state-machine model project for an observer device using AIclassifiers.

FIG. 15B depicts an exemplary Graphic User Interface (GUI) for editing,training, running, and refining a state-machine model for an observerdevice using AI classifiers.

The figures are provided to aid in the understanding of the inventionand their simplicity should not use to limit the scope of the invention.

Elements and acts in the figures are illustrated for simplicity andclarity, and the drawings have not necessarily been rendered accordingto any sequence or embodiment, and their simplicity should not use tolimit the scope of the invention.

DETAILED DESCRIPTION

In the following description, and for the purposes of explanation,numerous specific details are set forth in order to provide a thoroughunderstanding of the various aspects of the invention. It will beunderstood, however, by those skilled in the relevant arts, that thepresent invention may be practiced without these specific details. Inother instances, known structures and devices are shown or discussedmore generally to avoid obscuring the invention. In many cases, adescription of the operation is sufficient to enable one to implementthe various forms of the invention, particularly when the operation isto be implemented in software. It should be noted that there are manydifferent and alternative configurations, devices and technologies towhich the disclosed inventions may be applied. The full scope of theinventions is not limited to the examples that are described below.

In addition to the description provided above, additional information isprovided in the file history associated with this application and theassociated provisional application referenced above (e.g., as anAppendix to the Specification, Information Disclosure Statement,Transmittal Letter, etc.); those materials are hereby incorporated byreference in its entirety.

I. Hardware

Below, a general device and two different exemplary devices will bediscussed. One of the exemplary devices is a simple observer and theother is a more-complex observer.

These are meant to be two examples of observer devices of varyingcomplexity and do not necessarily constitute two different “embodiments”of the invention. Devices of varying complexity and forms, includingdevices that are composed of a combination of sensors described hereinbut not specifically detailed in an example, may work together as partof the system herein described. These devices may be less complex, morecomplex, or of intermediate complexity when compared to the exemplarydevices described.

I.1. Observer Hardware

With reference to FIG. 1, an exemplary observer 100 device isillustrated.

The observer 100 includes a computer or processor 102 (e.g., a RaspberryPi, Arduino, ESP32, iOS Device, etc.) that is coupled to or includesmemory 104, at least one wireless interface (such as a Wi-Fi interface106, Bluetooth interface 108, or other radio interface 110 (e.g., LoRA,ISM, etc.)), and a source of power 112 (e.g., battery or wall plug,wireless charging system such as one that uses the Qi standard, etc.).The observer may also be coupled to an auxiliary processor 114 (e.g., aneural network processor, graphics processing unit, math co-processor,etc.)

An observer might be an off-the-shelf voice assistant device (e.g.,Amazon Alexa, Google Assistant, etc.), an off-the-shelf camera device(e.g., Wyze Camera, RTSP camera, etc.) other commercially availableequipment, or custom purpose-built hardware.

The observer 100 may be coupled to one or more sensors, such as sensors1 . . . N 116. For example, an accelerometer (1-axis, 2-axis, or3-axis), gyroscope (1-axis, 2-axis, or 3-axis), magnetometer (1-axis,2-axis, or 3-axis), inertial measurement unit, temperature, humidity,pressure, voltmeter, ammeter, power meter, infrared signal receiver,fluid flow meter or detector, moisture detector, heart rate detector,GPS chip, light meter, distance sensor (infrared, ultrasonic, RADAR,LIDAR, etc.), passive infrared (PIR), or any other type of sensor etc.may be coupled to the observer 100. Any combination of theaforementioned sensors may be utilized. Furthermore, each sensor mayeither be directly or indirectly coupled to the observer 100 viawireless or wired technologies.

The observer 100 may also be coupled to one or more cameras or imagingdevices, such as cameras1 . . . N 118. For example, a visible lightcamera, an infrared camera or thermal camera (e.g., FLIR, etc.), a depthcamera (e.g., Kinect, etc.), a LIDAR device, etc. may be coupled to theobserver 100. Any combination of the aforementioned cameras may beutilized. Furthermore, each camera may either be directly or indirectlycoupled to the observer 100 via wireless or wired technologies.

The observer 100 may also be coupled to one or more sound devices. Forexample, a speaker or speaker array 120, a microphone or microphonearray 122, a mobile device or phone, a voice assistant (e.g.,Echo/Alexa, Google Assistant, etc.), or any other sound input or outputdevice, etc. may be coupled to the observer. Any combination of theaforementioned sound devices may be utilized. Furthermore, each sounddevice may either be directly or indirectly coupled to the observer 100via wireless or wired technologies.

The observer 100 may also be coupled to one or more output devices. Forexample, one or more output devices 124 such as a display (e.g, eInk,ePaper, OLED, TFT, etc.), light emitters 126 (e.g., spotlight, strobe,visible light, infrared light), haptic feedback device, a mobile deviceor phone, a voice assistant (e.g., Echo/Alexa, Google Assistant, etc.),or any other output device may be coupled to the observer 100. Anycombination of the aforementioned output devices may be utilized.Furthermore, each output device may either be directly or indirectlycoupled to the observer 100 via wireless or wired technologies.

The observer 100 may also be coupled to one or more input devices. Forexample, one or more input devices 128 such as a keyboard, a mouse, atouch sensor, a button, a mobile device or phone, a voice assistant(e.g., Echo/Alexa, Google Assistant, etc.), an indication from one ofthe other sensors described herein, or any other input device may becoupled to the observer 100. Furthermore, each input device may eitherbe directly or indirectly coupled to the observer 100 via wireless orwired technologies.

The observer 100 may also be coupled to one or more actuators, such asactuators 1 . . . N 130 or wireless actuators 1 . . . N 132. Forexample, one or more actuators such as a relay, infrared transmitter(receiver, or transceiver), power actuator (e.g., relay, power tail,wireless outlet controller controlled via Wi-Fi, Bluetooth, radio, etc.such as those made by Kasa, Switchmate, X10, etc.), motion actuators(e.g., servo motor, DC motor, stepper motor, linear actuator, etc.),sound actuators (e.g., speaker, buzzer, etc.), visible light actuators(e.g., LED (single or multi-color), strobe, display, lamp, etc.) or anyother actuator may be coupled to the observer 100. Furthermore, eachactuator may either be directly or indirectly coupled to the observer100 via wireless or wired technologies.

I.1.A Observer Hardware—Example Sensor-Based Observer.

As one example for purposes of illustration and without limitation, withreference to FIG. 2A, an observer 100 may be configured such as follows.

As one example, observer 100 includes at least a processor 202 (e.g., alow power Arduino device such as a device based on the Punch ThroughDesign Light Blue Bean), memory 204, a wireless interface 208 (e.g.,Bluetooth Low Energy, etc.), a source of power (e.g., coin cell battery)212, and a triple-axis accelerometer 210, and a display device 224(e.g., RGB LED).

As another example, observer 100 includes at least a processor 202(e.g., a processor such as the ESP32 chip such as a device based theTTGO T5 V2.2 development board), memory 204, a wireless interface 206(e.g., Wi-Fi, etc.), a source of power (e.g., USB port, LiPo ChargingCircuit, and LiPo battery) 212, and a triple-axis accelerometer 210(such as a GY521 or MPU6050 which also contains a temperature sensor atriple-axis gyroscope) and a display device 224 (e.g., eInk screen,LED(s), RGB LED, etc.). An exemplary schematic diagram for such anobserver is provided as FIG. 2B.

Either of the above examples may additionally or alternatively contain apiezo vibration sensor.

The user interface for either of the exemplary observer 100 devices canutilize at least any of the following: buttons built onto the device,the relative orientation of the device to the ground based on theaccelerometer or an IMU, RGB LED, a display, voice assistants, or mobiledevice applications, a website, etc. as further described herein.

The housing for either of the exemplary observer 100 devices isconfigured to allow the device to attach to an appliance or machine inorder to, for example, detect vibrations produced by the appliance whenthe appliance is running, and/or optionally detect movement of doorsthat are part of the appliance.

An exemplary case provides a way for an indicator or display such as anRGB LED or E-Ink display to be visible to a user of the device.Furthermore, the case may further include a magnet or an adhesive orsuction mount to enable it to removably attach to an appliance. Foradhesive or suction mounts, the case may further provide a rotatablebase that allows the observer 100 to rotate or flip relative to gravitywhile attached to the appliance. Any other mount may also be used withthe exemplary observer 100 device.

The observer 100 device is configured to detect whether it has beenrotated substantially with respect to the ground.

One way to do this is to periodically measure the acceleration vectorusing the accelerometer or other sensor and determine the largestcomponent of the acceleration vector. This can be used to determine ifthe observer 100 device has been rotated around a vector normal to theplane on which it is attached to the appliance (e.g., indicating thatthe user flipped it upside down), or if it has been rotated relative tothe horizon (e.g., indicating that the user opened a downward or upwardfacing door on a washing machine, dishwasher, or garage, etc.).

For example, with reference to FIG. 4, the orientation of the observer100 device with respect to the ground can be determined. Theaccelerometer, or other orientation sensor, in observer 100 providesacceleration components in the X, Y, and Z axis. Gravity is constantlyacting on the accelerometer with an acceleration of approximately 9.81m/s{circumflex over ( )}2. For an observer 100 device, this will be thelargest component of the acceleration of the device. Comparing the X, Y,and Z acceleration values provided by the accelerometer, the orientationof observer 100 with respect to the ground can be determined. If, at onetime, the observer reports that the largest acceleration component is inthe −Y axis, then at a subsequent time reports that the largestacceleration component is in the +Y axis, we can determine that thedevice has been rotated perhaps as much as 180 degrees. This can be usedas a user-interface input to, for example, change the state of theobserver device. In addition, by using trigonometry on the X, Y, and Zaxis acceleration components, the roll and pitch of the device can bedetermined. Based on this information, the angle of the observer 100device relative to the ground may be obtained. This information can beused, in conjunction with knowing the observer 100 is attached to a doorof an appliance, to determine when a user is interacting (e.g., loadingor unloading) the appliance, or to determine when a door is open (e.g.,a garage door).

Furthermore, when attached to an appliance or machine, the observer 100device may be configured to detect whether the appliance or machine isrunning by using the accelerometer.

With reference to FIG. 5, exemplary data collected by an observer 100including accelerometer-based data 502, temperature-based data 504,and/or gyroscope- or magnetometer based-data 506 is illustrated. Thedata further illustrates the orientation of the observer 100 relative tothe ground. For example, based on this information, an algorithm onobserver 100 can: determine when the door of an appliance was loadedbased on observer 100 device orientation changes 508; determine when theappliance 510 was running based on vibration or temperature changes;determine when the appliance completed running based on vibration ortemperature changes 512; and, determine when the appliance was unloadedbased on observer 100 device orientation changes 514.

In addition, an infrared detector, LIDAR, or other range or objectdetector, etc. can be incorporated into the device to determineproximity to another object such a block with adhesive or magnet placedon an appliance. This can be used, for example to determine if a door orwindow, that does not tilt up or down (and therefore does not change thegravity vector) has been opened. In one form of the invention, aninfrared emitter and detector are integrated into the observer. Theinfrared emitter transmits light which is then reflected off an adjacentobject. The infrared detector detects the light indicating that the dooror window is closed. If no reflected infrared light was detected, if itthe reflected light was below a threshold value, or if a LIDAR or otherrange finder indicates a larger than normal distance, then it wouldindicate the observed door, appliance door, or window was open.

With reference to FIG. 6A, data or statistics relating to the dataillustrated in FIGS. 5A, 5B, and/or 5C, can be used to train a neuralnetwork 602. To train the neural network, data is collected and taggedas illustrated in FIGS. 5A, 5B, and/or 5C. The neural network is thentrained using back-propagation techniques. Then, once the neural networkis trained, new data is provided to the neural network 602 as inputs604. Based on the input, the neural network then provides probabilityvalues at its output layer for each possible state 606. Thus, values offor the average energy across a series of acceleration measurements,along with orientation, temperature, and gyroscope readings can beturned into an output probability such as “closed,” “load,” “loading,”“loaded,” “running,” “unload,” “unloading,” “open,” etc. The highestprobability value, or the highest above a threshold can be taken as thecurrent state or used to drive a finite state machine model of anapplicable appliance.

Another way to determine if an appliance is running with observer 100 isto periodically measure the average energy of the accelerationcomponents over a short period of time using an accelerometer or othersensor. This may be done with a Fast Fourier Transform (FFT), or usingother methods. If the average energy over a short period of time exceedsa threshold amount that is determined to indicate the machine is runningbased on historical data, then the observer 100 device can determinewhether the appliance to which it is attached is running or idle.

For example, with reference to FIG. 6B, at act 620, observer 100periodically measures a sensor value such as the acceleration componentsover a short period of time (T) using an accelerometer or other sensor;and then performs a calculation on the collected data to obtain a resultsuch as a power spectrum density. The calculation may be done with aFast Fourier Transform (FFT), or using other methods to obtain anaverage FFT value for that data set. At act 622 if the reading of theresult of the calculation is greater than a threshold value, it can beinferred that the appliance is likely running 626 (or is likely in stateX, Y, etc.). If it is not, it can be inferred that the appliance islikely not running 624 (or not in state X, Y, etc.). At act 628, if itis determined the appliance is likely running this is noted in anobservation history. At act 630, if it is determined that the past M(e.g., an integer of 1 or more) measurements indicate the machine isrunning (or in state X, Y, etc.), then it can be inferred or determinedthat the machine is running (or in the “likely” *M state X, Y, etc.).The observer may sleep or delay by a period of time D at act 632, beforetaking another measurement.

Other algorithms may also be used. For example, the standard deviationof measurements in any of the X, Y, and/or Z axis can be compared to athreshold value. If the value is exceeded, then it is inferred that theappliance is running.

An IMU utilizing an accelerometer, gyroscope, and magnetometer, or anyother device, may also be used to detect orientation and vibrationchanges.

Either of the exemplary observer 100 devices may either broadcastinformation using Bluetooth Low Energy or connect to remote devicesthrough Bluetooth or Wi-Fi as further described below. See Source Tables4, 5, 6.

Artwork-Based e-Paper State Display. In some forms of the invention, anelectronic display (e.g., an LCD, LED, OLED, E-Ink, projection orprojector display, or any other type of electronically controlleddisplay, etc.) may be utilized.

In some forms of the invention, for example, instead of the digital (oranalog) display including textual information such as “LOAD”, “RUNNING”,“DONE” etc., artwork can be used.

For example, predominantly dark graphics or text can be used for“LOAD”/“DIRTY”, and predominantly light graphics or text can be used for“DONE”/“CLEAN.” Colors may also be used and associated with states. Forpurposes of illustration and without limitation, if a color e-paperdisplay (e.g., red/black/white, full color, etc.) is used, then red maybe used for one of the states such as “LOAD”/Dirty. Of course, anyassociation between state, color, and/or brightness might be utilized.

As an additional example, in some forms of the invention, the graphicsmay be line art or a picture that a user uploads and associates with theobserver device or appliance. The artwork, graphics, text, or otherinformation to be displayed may also update periodically from a server.

The artwork can also be used in conjunction with a textual descriptionof the state by having the textual description superimposed or overlaidon top of the art.

Pinwheel or Mechanical Display. In some forms of the invention, amechanical display may be utilized. For example, E-ink and otherelectronic displays can be relatively expensive. Instead of or inaddition to using these displays, a motor or servo with a pinwheel,flap, or other mechanism or object may be used. The object may includedifferent colors or messages on its surface that can rotate, flip, orotherwise move as needed while the observer device is active (i.e., notin a low electrical current sleep mode, etc.) and then persistentlydisplay while the device is inactive (i.e., in a low electrical currentsleep mode).

Projection Display. In various forms of the invention, a projector (e.g.projector, pan/tilt projector, spherical projector, etc.) may be used toproject information onto observed machines. The projector may brightenand dim based on ambient light amount, user preferences, or time of day,and may project state above device at user eye level, and may move sothat when appliance is in use or blocked by the user the annotation isnot occluded and is still visible to the user nearest the appliance ormachine. Alternatively, or in addition, the information may also be datasent to a server that is then used to annotate an augmented realitydevice instead of physically projecting light onto the device Forpurposes of illustration and without limitation, if a kitchen dishwasheris observed, when it is ready to load a red “Load” state may beprojected onto the appliance; if running a blue “Running” statedesignator could be projected onto the device; if done a green “Done”designator may be projected onto the appliance or above the appliance.As another example, if an observed stove is hot, then a red hotindicator can be projected onto or above the appliance as to alert usersthat the device is hot.

I.1.B Observer Hardware—Example Vision-Based Observer.

As one example for purposes of illustration and without limitation, withreference to FIG. 2C, an observer 100 may also be configured as follows.Observer 100 includes at least a processor 202, memory 204, a wirelessinterface 206 (e.g., Wi-Fi, etc.), a source of power 212 (e.g., wallplug), and a camera 218. Such devices may be existing third-partysecurity camera devices (such as those produced by Ring, Nest, Zmodo,Wyze, any Real Time Streaming Protocol (RTSP) compatible camera, etc.).The images recorded from those devices may be obtained using anApplication Programming Interface (API), and analyzed or processed on aremote computing device.

As another example, observer 100 includes at least a processor 202(e.g., a Raspberry Pi 3B+), memory 204, a wireless interface(s) 206 and208 (e.g., Bluetooth Low Energy, Bluetooth, Wi-Fi, etc.), a source ofpower 212 (e.g., wall plug or battery), and a camera device 218 (e.g., aPiCamera, etc.).

As another example, observer 100 includes at least a processor 202(e.g., a Raspberry Pi 3B+), memory 204, a wireless interface(s) 206 and208 (e.g., Bluetooth Low Energy, Bluetooth, Wi-Fi, etc.), a source ofpower 212 (e.g., wall plug or battery), a camera device 218 (e.g., aPiCamera, etc.) and optionally an audio capture device 222 (e.g., USBMicrophone, etc.).

As another example, observer 100 includes at least a processor 202(e.g., a Raspberry Pi 3B+), memory 204, a wireless interface(s) 206 and208 (e.g., Bluetooth Low Energy, Bluetooth, Wi-Fi, etc.), a source ofpower 212 (e.g., wall plug or battery), and a camera and depth cameradevice 218 (e.g., a Kinect, etc.).

These examples may be modified and adapted to different, substitute, orequivalent components based on availability and for design formanufacture.

The user interface for any of the exemplary observer 100 devices canutilize at least any of the following: buttons built onto the device,the relative orientation of the device to the ground based on theaccelerometer or an IMU, RGB LED, a display, voice assistants, or mobiledevice applications, a website, etc. as further described herein.

The housing for any of the exemplary observer 100 devices is configuredto allow the camera to be directed to view an appliance in order tocapture images (e.g., RGB, depth, thermal, etc.) and sound (e.g., toproduce a spectrogram) produced by the appliance and/or peopleinteracting with the appliance. For example, sequential images and soundrecordings of users loading, starting, and unloading the appliances aswell as sequential images and sound of the device running can be used totrain one or more neural networks to infer the current state of adevice.

These exemplary observer 100 devices may simply capture images and soundfor further processing on a backend server, or the observer 100 devicesmay utilize onboard processors (including optional neural networkprocessors) to analyze the captured information themselves.

The observer 100 device may alternatively be integrated into orinterfaced with (wired or wirelessly) an appliance. For example,observer 100 device may be interfaced with a control circuit of anappliance in order to determine or infer state information of theappliance. Observer 100 may be configured to have its camera face theappliance from a remote location, or may be integrated with theappliance as to be able to observer users that interact with theappliance. Observer 100 may perform facial recognition on the users inorder to determine or infer not only the state information of theappliance, but also the last person to load or run the appliance. Upondetecting the appliance has finished running, observer 100 may thennotify the recognized individual to unload the appliance. See SourceTables 7 and 8.

I.1.C Observer Hardware—Display Features.

In addition to acting as an observer, observers attached/connected/incommunication with a display may also act as a smart screen device thatdisplays information such as weather information, spelling information,conversion information, or timer information. This information may beobtained directly from the internet or may be obtained or pushed from aconnected or nearby voice assistant.

For example, the observer may display information it obtains from theinternet. At a specific time of day or when the observer has been idlefor a period of time, it may show: (1) only the weather forecast; (2) asplit screen with the appliance state and a weather forecast; and/or,(3) a weather forecast with appliance state also shown somewhere on thescreen. The weather forecast shown can be one that is based on apre-configured setting or a location determination or setting.

For example, if a user queries a voice assistant for the weather, thevoice assistant may also transmit a graphic display of the weatherforecast to a nearby observer device if the observer is equipped with ascreen.

For example, if a user queries a voice assistant to start a timer, thevoice assistant may also transmit a graphic display of the timercountdown, or an instruction to graphically display a timer countdownending at a certain time, to a nearby observer device if the observer isequipped with a screen.

In either case, the voice assistant may transmit the instructions orinformation via any wireless communication channel such as Wi-Fi, BLE,etc.

I.1.D Observer Hardware—Example Sensor- and Vision-Based ObserverEnclosure.

Different case designs can be used with an observer 100 device dependingon the device capabilities.

For example, with reference to FIG. 2D, one form of a sensor-basedobserver 100 device is depicted. The case 230 contains a clear window234 for status led diffusion to facilitate viewing from any angle.Similarly, FIG. 2E provides another form of a sensor-based observerdevice with case 232 and clear window 234. The status led may indicatethe state of the device with a constant or intermittent colorillumination. For example, a red light may indicate “Dirty/Ready toLoad,” a green light might indicate “Clean/Done,” a blue light mayindicate “Washing/Running,” a yellow light may indicate “Alert/Caution,”a white light may indicate a system status such as booting up, firmwareupdating, Wi-Fi connected, etc. The case 230 and/or 232 may also haverounded corners, a teardrop cross section, and/or a base level ofthickness so no sharp edges on hypotenuse. The case 230 and/or 232 mayalso have any other shape. The case may have a magnetic, adhesive orother attachable and/or detachable backing for application of the caseto an observed appliance or machine.

For example, with reference to FIG. 2F, one form of an image-basedobserver 100 device is depicted. The device contains an imaging device236, a projection device 238, and optionally a mount 237. The imagingdevice 236 may be any type or combination of imaging devices. Thevarious forms of the image-based observer 100 device may also containsensors for non-imaging-based classification. The imaging device 236 maybe able to pan and/or tilt to change the field of view either manuallyor under electronic control. The projection device 238 may be any typeof projector and may project the current status 239 of the observedobject onto or within the proximity of the observed object. Theprojected current status 239 may be placed higher or lower and/or to theleft or right of the projection area to better correlate the status withthe observed device in the real world. In some forms of the device, themount 237 is present. The mount 237 may be manually controlled or may beelectronically controlled in pan/tilt/angle and/or any other axis.

For example, with reference to FIG. 2G, another form of a sensor-basedobserver 100 device case is depicted. The case has a front portion 242,and a back portion 240. With reference to FIGS. 2H and 2I, asensor-based observer 100 device including a display (e.g., a 2.9-inchE-ink, etc.) is depicted. The device also optionally includes buttons, aUSB charging port, an LiPo battery (e.g., 18650 size, etc.), RGB LED,Red LED, Green LED, etc. The sensor-based observer device can be screwedinto the back portion 240 of the case. As depicted in FIG. 2J, the frontportion 242 of the case snaps over the front of the device into the backportion 240 of the case while leaving the display 244 visible. If thesensor-based observer did not have a display, or had a different type orsize of display, an alternate front portion 242 of the case may beprovided with an appropriate opening or no opening. Furthermore, othershapes of front portion 242 may be provided for fun (e.g., like arefrigerator or appliance magnet, etc.), aesthetics (e.g., differentcolors, designs, etc.), branding (e.g., detergent company, soap company,dryer sheet company, etc.), etc. As depicted in FIG. 2K, the back mayhave an adhesive or magnetic backing 246 that allows the observer 100device and/or case to attach and/or detach from an observed appliance ormachine.

Observer devices with a display (e.g., E-ink, etc.) may update thedisplay in order to provide information to people passing by theobserver device and the observed object. For purposes of illustrationand without limitation, with reference to FIG. 2L, the observer devicedisplay has been updated to indicate that a Dishwasher appliance towhich it is (removably) attached is “Done.” The screen also indicatesthat the device may be “flipped” (e.g., rotated 180 degrees upside down)to transition the state of the observer from “Done” to “Load” (e.g.,“Flip→Load”, etc.). The device may also provide additional informationsuch as a graph showing the “FFT. Avg.” value detected by the observerdevices. In this particular example, the reduction in the “FFT. Avg.”over time to a sustained level below a threshold value has caused theobserver to conclude that the observed dishwasher is now “Done.” Thedone indication may be placed on a light color background with dark textso that it is easily visible from a distance.

For purposes of illustration and without limitation, with reference toFIG. 2M, the observer device display has been updated to indicate that aDishwasher appliance to which it is (removably) attached is “Running.”The device may also provide additional information such as a graphshowing the “FFT. Avg.” value detected by the observer devices. In thisparticular example, the increase in the “FFT. Avg.” over time to asustained level above a threshold value has caused the observer toconclude that the observed dishwasher is now “Running.” The runningindication may be placed on a dark color background with light text.

For purposes of illustration and without limitation, with reference toFIG. 2N, the observer device display has been updated to indicate that aDishwasher appliance to which it is (removably) attached is “Ready toLoad.” The screen also indicates that the device may be “flipped” (e.g.,rotated 180 degrees upside down) to transition the state of the observerfrom “Load” to “Done” (e.g., “Flip→Done”, etc.). The device may alsoprovide additional information such as a graph showing the “FFT. Avg.”value detected by the observer devices. In this particular example, theobserver has concluded that the observed dishwasher is now ready to“Load,” because it was either flipped 180 degrees while displaying“Done” or because it was rotated downward such that the screen wasfacing the ground for a period of time to indicate that the user had thedishwasher door open for a sufficient period of time to indicate thatthe dishwasher was unloaded after it was “Done.” The load indication maybe placed on a dark color background with light text so that it iseasily visible from a distance. Alternatively, the load indicatingbackground may be a dark red or a yellow or any other color depending onthe capabilities of an E-Ink or other type of display.

Power Management. In various forms of the invention, the battery-poweredobserver device manages its power by entering deep sleep mode forvarying amounts of time, as shown in the various Source Tables, then theobserver wakes up and takes a measurement with its sensors. If no actionis needed, the observer re-enters deep sleep; otherwise, information isupdated on the screen of the device (if applicable) and on the serverbackend via the Internet as necessary.

I.2. System Overview.

Below, a general system and different exemplary systems will bediscussed. These are meant to be examples of systems of varyingcomplexity and do not necessarily constitute two different “embodiments”of the invention. Systems of varying complexity and forms, includingsystems that are composed of a combination of devices and sensorsdescribed herein but not specifically detailed in an example, may worktogether as part of the system herein described. These devices may beless complex, more complex, or of intermediate complexity when comparedto the exemplary devices described.

I.2.A Example Acceleration or Sensor-Based Observer System(s).

With reference to FIG. 3A, the system includes at least one observer 100device such as devices 300, 302, and 304.

The system may also include a mobile phone 306 with a Wi-Fi interface312, Bluetooth interface 310, and a cellular interface 308 capable ofestablishing a connection to the Internet 320. The mobile phone 306 mayfurther contain a control application 314 configured to specificallyinterface with observer 100 devices.

The system may also include any of the following: a Wi-Fi access pointand/or router 316 coupled to a modem 318 capable of establishing aconnection to the Internet 320, a radio hub 322 (e.g., a device with anycombination of a Bluetooth, LoRA, or other radio interface running aMQTT or other server, etc.) coupled to the Wi-Fi access point, a smarthome device 300 coupled to the router 316 and a voice assistant 324wirelessly coupled to the Wi-Fi access point.

The system may be coupled, via the Internet 320, to a voice assistantbackend 326 (e.g., Amazon Alexa, Google Assistant), an observer backend328 (e.g., Amazon AWS, Microsoft Azure, Google Cloud, Blynk, etc.), anda push notification backend (e.g., Apple, Android, Twitter, SMS, etc.).

I.2.A.i BLE Device Paired to Phone

As one example implementation of the system, an observer 100 utilizes aBluetooth Low Energy (BLE) interface broadcast data to a nearby mobiledevice 306. The mobile device 306 may then relay this information to aback-end server 328 via the Internet 320.

As another example, when an observer 100 device is idle it may act as aniBeacon so that it can awaken mobile devices 306 that pass by theobserver 100 and communicate with them. The iBeacon broadcast may alsocontain coded information relating to the inferred state of the observedappliance as part of the major ID or minor ID of the iBeacon broadcast.The mobile device 306, upon receiving an iBeacon or BLE advertisement,may respond by pairing or initiating a communication session over theBluetooth interface to exchange data with the observer 100. The observer100 may also collect information from its sensors when idle.

The mobile device 306 may also check to see if there is updated data onthe back-end server 328 that is to be sent to the observer 100 device.If there is such data, then the mobile device will retrieve the data viaits Wi-Fi or cellular interface over the Internet and then transmit thatdata to the observer 100 over the Bluetooth interface.

The observer may then hibernate while it optionally collects more sensordata for a period of time, or it may immediately become an iBeacon orBLE advertiser again.

I.2.A.ii Phones as Data Aggregators Using Bluetooth Beacon Transmissions

As another example, mobile device 306 has an application 314 that isinstalled that listens for all UUIDs assigned to observer 100 devices.Currently iOS allows an application to listen for 20 different UUIDs.Each UUID could have 0xFFFF unique major IDs and 0xFFFF unique minorIDs. When a mobile device 306 senses one of the UUIDs, it can startranging for beacons and obtain the major and minor IDs and report thatto a central server.

A back-end server 328 can process that information, associate theinformation with a specific observer 100 devices, determine the state ofthat observer 100 device based on the encoding in the major and minorIDs that indicate what device it is, and what its current state is.

The back-end server 328 can then notify appropriate user's that havesubscribed to notifications for state changes from that device. In thisway, third parties that pass by an observer 100 may relay informationabout the state of the observer to interested parties without having topair the observer 100 to the mobile device 306 (which would beimpractical for third parties to do for unknown-to-them observer 100devices).

I.2.A.iii BLE Device Paired to HUB

As another example, observer 100 devices may be paired via Bluetooth orBluetooth Low Energy or any other communication technology to a radiohub 322.

Radio hub 322 may be a device such as a Raspberry Pi running a NodeRedMQTT server. The data hub 322 is connected via Wi-Fi or wired connectionto the home router 316. The data hub can then orchestrate the data flowfrom the observer 100 to other services and actions around the house.

Radio hub 322 may also be a device such as a Google Home or Amazon Echosmart speaker. The observer 100 may communicate with Bluetooth to theradio hub 322, which then performs additional actions such as notifyinga phone or relaying information from the observer 100 to a back-endserver for further processing and/or notification.

I.2.A.iv Observer Connected Via Wi-Fi

As another example, observer 100 devices include a Wi-Fi interface andconnect to a home Wi-Fi network associated with a user. The devices areable to connect through the Wi-Fi router 316 and modem 318 to theInternet 320 and update the observer back-end database 328. The observerback-end database may be any back-end service including for example thatprovided by Blynk.io (or AWS, Azure, etc).

A portion of the observer back-end 328, or any other part of the system,may also store firmware updated 330 for observer devices 100 such asdevices 300, 302, and 304. Observer devices 100 may connect to theInternet 320 to request any available firmware updates 330 for theirparticular hardware revision if the firmware version is greater than thecurrent firmware version on the observer device. Alternatively, firmwaremay be pushed to the observer devices 100.

I.2.B Example Vision-Based Observer System(s).

With reference to FIG. 3B, the system includes at least one observer 100device such as devices 1 through N 350, 352, 354, and 356. At least oneobserver 100 is configured to visually observe at least one machine suchas machines 1 through M 360, 362, 364, 366, and 368. The exemplarysystem of FIG. 3B may be combined with the system depicted in FIG. 3A orany other disclosure in this specification.

The system may also include at least one mobile device(s) 306 with atleast a Wi-Fi interface, Bluetooth interface, and optionally a cellularinterface capable of establishing a connection to the Internet 320. Themobile device(s) 306 may further contain a control application 314configured to specifically interface with at least one of observer 100devices.

The system may also include any of the following: a Wi-Fi access pointand/or router 316 coupled to a modem 318 capable of establishing aconnection to the Internet 320, a radio hub 322 (e.g., a device with aBluetooth, LoRA, ZigBee, or other radio interface running a server suchas an MQTT server, etc.) coupled to the Wi-Fi access point, a smart homedevice coupled to the router such as a Phillips Hue light controller andat least one voice assistant or smart speaker such as smart speakers 1through N 324 wirelessly coupled to the Wi-Fi access point.

The system may be coupled, via the Internet 320, to a voice assistantbackend 326 (e.g., Amazon Alexa, Google Assistant), an observer backend328 (e.g., Amazon AWS, Microsoft Azure, Google Cloud, Blynk, etc.), anda push notification backend (e.g., Apple, Android, Twitter, Email, SMS,etc.).

Observer 100 devices of different types and capabilities (includingvision-based sensing) can be combined in any system. Furthermore,accelerometer-based and vision-based observers can be combined in thesame system.

For example, a first device 350 is an observer 100 device that mayinclude a camera, or a camera and microphone. It is configured toobserve a dishwasher 360. The first device 350 observes the dishwasher,and notes when it is ready to load/dirty, running, or done/clean andneeds to be unloaded. The first device 350 may accomplish this byprocessing locally or remotely any combination of images, sounds,videos, etc. to determine current state and/or state change events thatit records to either the radio hub 322, the observer backend 328, orboth. The first device 350 may also be configured to recognize theidentity of the person who last loaded dishwasher 360 in order to sendtargeted notifications to that person when the dishwasher is done orwhen it is ready to be unloaded.

As another example, a second device 352 is an observer 100 device thatmay include a camera, or a camera and microphone along with an infraredtransceiver/transmitter 353. It is configured to observe atelevision/entertainment center. The second device 362 may be configuredto recognize when the television has been neglected—that is wheneveryone left the room without turning off the TV. In such instances,the second device 352 will automatically turn the television off viatransmitting a command from infrared transceiver/transmitter 353 toinfrared transceiver/receiver 363. The second device 352 may also beconfigured to recognize when commercials are played on television and toautomatically mute the television during commercials, or to fast forwardor skip ahead until non-commercial content is detected on thetelevision.

Still further, an exemplary third device 354 is an observer 100 devicethat may include a camera, a thermal camera, and a microphone. The thirddevice is configured to observe an oven 364. The thermal image may beused to determine if the oven has been neglected in an “On” state.

In the illustrative example, a Nth device 356 is an observer 100 devicethat may include a camera or a camera and microphone along with aninterface, either wired or wireless, to an actuator such as a power ormotion actuator 370. The fourth device may be configured to observemultiple machines such as a washing machine and a dryer 366. It may alsobe configured to observe another machine such as a hot water heater 368.The fourth device is configured to track the state of the washingmachine and the dryer and communicate the updated state as appropriateto either the radio hub 322, the observer backend 328, or both. It isalso configured to detect an abnormal state with the hot water heater368. If an abnormal state is detected with the hot water heater 368,then the actuator 370 can shut off power to the hot water heater (if itis a power actuator) or turn off the water supply to the hot waterheater (if it's a motion actuator) and alert the appropriate person orpeople. Different actuators can be hooked to different machines andcoupled wirelessly to the observer system in order to go beyond merelymonitoring devices and also impart change on the observed devices.

These observer 100 devices 350, 352, 354, and 356, are configured totransmit updated state or status information or commands via the Wi-Ficonnection. These commands are routed to the Internet. Updated stateinformation is transmitted to either the radio hub 322, the observerbackend 328, or both. Commands to actuate a power actuator or lightcommand may be routed to the internet to a backend service such as avoice assistant backend 326, and thereafter a response is received. Avoice query is routed from smart speakers 324 via Wi-Fi accesspoint/router 316, modem 318, and via the Internet 320 to voice assistantbackend service 326. Voice assistant backend service may communicate(transmit information, receive information, or both) with observerbackend 328 via the Internet 320 to receive updated state information orissue commands to the observer backend. The observer backend may relaythose commands back to the appropriate observer devices via the Internet320. Voice assistant backend service 326 formulates a voice responsewith retrieved information and causes the voice assistant 324 to audiblyrespond. See Source Tables 7 and 8.

I.2.C Example Other System(s).

The systems may be interfaced with various other systems such asnotification systems (e.g., IFTTT, Apple Push Notification System,Android Push Notification System, SMS Notification Systems, Twitter,Amazon Alexa Voice Announcement System, Google Home Voice AnnouncementSystem, etc.), home lighting systems (e.g., Phillips Hue, etc.), andsmart plug systems (e.g., Kasa, WeMo, etc.), or any other system thathas an Application Programming Interface or other access mechanism.

II. State Machines

Inferring the state of an appliance attached to or observed by observerdevice 100 may be accomplished using a finite state machine model of theappliance.

II.1 Example State Machines

For purposes of illustration and without limitation, the various formsof the observer 100 devices may be utilized with state machine typesdescribed herein to model the behavior of appliances that may beobserved with the observer 100.

Two-State Machines. Machines that have a first state and a second state.See, e.g., Table 1, below.

Three-State Machines. Machines that have three states. See, e.g., Table2, below.

More-State Machines. Machines that have an integer number of statesgreater than three.

Generic Machine or Abnormal State Detector. Generic machines that have anormal state and an abnormal state. (e.g., Hot water heater, Saw, etc.).See, e.g., Tables 3.1 and 3.2, below.

Neglectable Machine or Openable Machine. Machines that users may forgetto return to non-used state. For example, garage doors, ovens, stoves,refrigerators, sinks, TVs, lights, etc. See, e.g., Tables 4.1 and 4.2,below.

Single User Machine. Machines that can be occupied or un-occupied (e.g.,Rooms such as bathrooms or conference rooms). See, e.g., Tables 5.1 and5.2, below.

Loadable Machine. Machines that are loaded, run, and then complete acycle. (Washer, Dryer, Dishwasher, Coffee Maker, Microwave, etc.). See,e.g., Tables 6.1 and 6.2, below.

Cyclic Machines. Machines that are intended to cycle from an idle stateto a working state repeatedly without issue. See, e.g., Tables 7.1 and7.2, below.

The above categories of state machines are for illustration only and maybe modified in the claims simplify or combine various aspects orterminology used. Furthermore, the more detailed descriptions of each ofthe general categories of state machines may be modified to add orremove states and transitions to simplify or add features.

Each of the below tables are applicable to any type or configuration ofobserver such as those relying on various sensors, accelerometer data,vision data, sound data, etc. The notation used in Tables 1 through 7 isas follows: (A) each state is denoted in brackets; for example, State:[Name of State] (B) each transition is denoted with an arrow; forexample, Transition to State:→State; (C) each line in each cell isoptionally independent of the other lines in the cell. For example,“Detected [1]. (newline) Detected [2]. (newline) Detected [3].” shouldbe read as “[1] alone, [2] alone, [3] alone, or any combination of [1],[2], and [3] (and possibly other factors).”

TABLE 1 Two-State Machine. Machine [State 1] → State 2 [State 2] → State1 Generic/Other State 1 Detect specified activity 2. State 2 Detectspecified activity 1. Detect rotation of device Detect rotation ofdevice around around specified axis. specified axis. Dishwasher Ready toDetect running done with Ready to Detect unloaded (e.g., door open forLoad. accelerometer and/or Unload. greater than threshold time, etc.).Dirty. temperature. Clean. Detect rotation of device around Detectrotation of device specified axis. around specified axis. WashingAvailable. Detect running done with Ready to Detect unloaded (e.g., dooropen for Machine Ready to accelerometer and/or Unload. greater thanthreshold time, etc.). (Top Load) Load. temperature. Done. Detectrotation of device around Load. Detect rotation of device specifiedaxis. around specified axis. Washing Available. Detect running done withReady to Detect unloaded (e.g., door open for Machine Ready toaccelerometer and/or Unload. greater than threshold time, etc.). (FrontLoad) Load. temperature. Done. Detect rotation of device around Load.Detect rotation of device specified axis. around specified axis. DryerAvailable. Detect running done with Ready to Detect unloaded (e.g., dooropen for Ready to accelerometer and/or Unload. greater than thresholdtime, etc.). Load. temperature. Done. Detect rotation of device aroundLoad. Detect rotation of device specified axis. around specified axis.Garage Door Open Detected closed (e.g., Closed Detected open (e.g.,rotation of 90 rotation of 90 degrees degrees around specified axis.)around specified axis.)

TABLE 2 Three-State Machine. Machine [State 1] → State 2 [State 2] →State 3 [State 3] → State 1 Generic/ State 1 Detect specified State 2Detect specified State 3 Detect specified activity Other activity 2.activity 3. 1. Detect rotation of Detect rotation of Detect rotation ofdevice device around device around around specified axis. specifiedaxis, specified axis. Washing Available. Detect running Running. Detectrunning Ready to Detect unloaded (e.g., Machine Ready to with Washing.complete with Unload, door open for greater Load. accelerometeraccelerometer Done, than threshold time, etc.). Load, and/or and/ortemperature. Detect rotation of device temperature. Detect rotation ofaround specified axis. Detect rotation of device around device aroundspecified axis. specified axis. Dryer Available. Detect running Running.Detect running Ready to Detect unloaded (e.g., Ready to with Drying.complete with Unload, door open for greater Load, accelerometeraccelerometer Done. than threshold time, etc.). Load, and/or and/ortemperature. Detect rotation of device temperature. Detect rotation ofaround specified axis. Detect rotation of device around device aroundspecified axis, specified axis. Dish- Ready to Detect running Running.Detect running Ready to Detect unloaded (e.g., washer Load, withWashing. complete with Unload. door open for greater Dirty.accelerometer accelerometer Clean. than threshold time, etc.). and/orand/or temperature. Detect rotation of device temperature. Detectrotation of around specified axis. Detect rotation of device arounddevice around specified axis, specified axis.

TABLE 3.1 Generic Machine/Abnormal State Detector Specific Machine[Normal] → Alert [Alert] Normal Generic/Other Normal. Problem detected.Alert Problem resolved, appropriate or alert dismissed person or people.Water Heater Normal with no water on ground. Problem detected. AlertNormal situation (observer device Normal with no moisture detectedDetect rotation of appropriate detected with attached to the on ground.device around person of nature water heater. body of the heater) Normalwithin specified specified axis, of problem. temperature range. Detectabnormal temperature reading. Toilet Normal operation. Abnormaloperation Alert Toilet normal (observer device Flowing water detectedfor detected (e.g., appropriate operation attached to pipe or expectedperiod using constantly running person or detected. fixture)accelerometer, vibration, or water water, water detected people. flowsensor. If equipped, no on floor, etc.) moisture detected on floor neartoilet using moisture detector. Person/Baby Asleep or not crying.Detected awake Alert Detected asleep (e.g., Awake, Vibration detectedassociated with and/or crying. appropriate and/or not crying. Crying,etc.) the baby being awake. person or Sound detected associated withpeople. the baby crying. Person Not pulling hair. Detected motion AlertDetected motion (e.g., monitor Not exhibiting motion associatedassociated with appropriate associated with trichotillomania withdisorder based on disorder. person or disorder has (hair pulling) oraccelerometer worn on wrist, etc. people. stopped. other disorder.)Person (e.g., Person standing, sitting, laying Detected person falling/Alert Detected situation monitor not down on bed. Person not laying orlaying hunched appropriate resolved. fallen). down or hunched over onfloor over where not person on based on accelerometer reading expected(e.g., not at people. from wrist, GPS position home based on GPS,Optionally, alert indicating a public place, heart while wearing theemergency rate or body heat indicating the device on wrist). services.device is being worn.

TABLE 4.1 Neglectable Machine Specific Machine [Normal] → Vulnerable[Vulnerable] → Alert [Alert] → Normal Generic/ Normal. Detected the Themachine Determines that the Alert Detected Other machine has is in amachine has been appropriate machine placed been placed in vulnerableneglected while in person or in normal state, a vulnerable state. thevulnerable state people. or alert state. dismissed Refrigerator. Not inuse Detected door Door open Detected prolonged Alert Detected door(observer with door open based on while in use door open stateappropriate closed. device closed. orientation Detected warning personor attached to reading beeping using people. door) microphone fromrefrigerator for prolonged time without user interaction. Stove Not inuse Detected Heating or hot. Detected prolonged Alert Detected coolingwith door heating or hot vulnerable appropriate using closed. usingDetected timer person or temperature temperature going off using peoplesensor. sensor or microphone for remote prolonged time temperaturewithout user sensor. interaction. Oven Not in use Detected Steady stateDetected steady Alert Detected oven (observer with door heating or hotheating or hot, state heating or hot appropriate off or cooling deviceclosed. using for prolonged person or and door closed. attached totemperature period using people. door) sensor, temperature sensor.Detected timer on oven beeping for prolonged time without userinteraction. Detected door open and close with accelerometer withoutcooling period following based on temperature sensor (e.g., user forgotto turn off the oven). Sink Not in use/ Detected Running. Detectedprolonged Alert Detected water (observer off based on running based onstate based on appropriate turned off based device vibration onvibration vibration and/or person or on vibration attached to and/orand/or temperature people. and/or pipe or temperature temperaturereadings temperature fixture) readings readings. readings. TelevisionNot in use/ Detected on On. Detected prolonged Emit Detected (lightsensor off. based on light on state while not infrared machine placedplaced on sensor placed near a human based signal to in normal state,power led, over power on motion sensor. turn off the or alert motionled. television, dismissed detector Alert facing appropriate outwardfrom person or tv) people. Curling Iron Not in use/ Detected Steadystate Detected prolonged Remove Detected (accelerometer off. heating orhot, heating or hot steady state heating power from machine placed inhandle and while in use or hot while not in the :: in normal state,either a temp by person use by a human machine, or alert sensor or basedon based on Alert dismissed. power usage accelerometer accelerometerdata. appropriate Detected sensor) data. cooling. person or Detectedpower people. off. Garage Door Closed Detected open/ Open. Detectedprolonged Alert Detected closed/ opening period of time appropriateclosing based on based on open person or accelerometer accelerometerpeople. data. data.

TABLE 5.1 Single User Machine Specific Machine [Available] → In Use [InUse] → Alert [Alert] → Available Generic/ Machine is Detected that theMachine is A problem is Alert appropriate Detected that the Otheravailable. machine has occupied detected with person or people, machinehas become occupied the machine or and/or note that become it is flaggedas the machine is unoccupied. (If out of order. not available from[Alert], the because it is out alert flag is reset) of order. Room Roomis Detected human Room is Room flagged Alert appropriate Detected human(observer not entering the room occupied. as out of person or people.leaving the room mounted on occupied. using a motion service. Note thatthe using a motion door facing sensor. room is not sensor. inward)Detected human available because Detected human entering the room it isout of order, leaving the room using an using an accelerometer oraccelerometer. other sensor If from [Alert], the discussed herein alertflag is reset). Parking Parking Detected car Car is A problem is Alertappropriate Car observed Spot spot is entering the occupying detectedwith person or people. leaving the spot (observer empty. parking spot orthe parking the parking spot Note that the using a distance mountedoccupying the spot. or it is flagged machine is not sensor. aboveparking spot using as out of order available because parking a distancesensor or reserved. it is out of order. spot) or other sensorOptionally, a discussed herein. non-car object is detected to be in thespace in the space. Seat Seat is not Detected object or Object or Aproblem is Alert appropriate Person observed (mounted in occupied. humanoccupying human detected with person or people. leaving the seat seat oron the seat based on a occupying the seat or it is Note that the basedon pressure seat) pressure sensor the seat. flagged as out machine isnot sensor reading. reading or other of order or available becausesensor discussed reserved. it is out of order. herein. Detect that anobject is in seat, alert if in seat too long. Detect that the seat hastipped over based on accelerometer reading.

TABLE 6.1 Loadable Machine Specific Machine [Load] → Loaded [Loaded] →Running [Running] → Alert [Alert] → Unload [Unload] → Load Generic/Machine Detected . Machine Detected Machine Detected There is DetectedMachine Detected Other is user is loaded machine is loaded problem athat the has user empty loading and running and or error problem machinecompleted unloading and not machine ready to running or error hasrunning. machine. in use. run. Stop the completed machine its immedi-running ately. state Alert without appropri- error. ate people. WashingAvailable Detected Machine Detect Running. Detected There is DetectReady to Detect Machine Ready user is loaded running Washing problem arunning Unload. unloaded to loading and with or error problem completeDone. (e.g., Load machine ready to accel- or error with door Load run.erometer Stop the accel- open for and/or machine erometer greatertemper- immedi- and/or than ature. ately. temper- threshold Detect Alertature time, rotation appro- Detect etc.). of priate rotation Detectdevice people of rotation around device of device specified aroundaround axis. specified specified axis. axis. Dryer Avail- DetectedMachine Detect Running. Detected There is Detect Ready to Detect ableuser is loaded running Drying problem a running Unload. unloaded Readyloading and with or error problem complete Done. (e.g., to machine readyto acceler- or error with door Load run. ometer Stop the acceler- openfor Load. and/or machine ometer greater temper- immedi- and/or thanature. ately. temper- threshold Detect Alert ature. time, rotationappro- Detect etc.). of priate rotation Detect device people. ofrotation around device of device around around specified specifiedspecified axis. axis. axis. Dish- Ready Detected Machine Detect Running.Detected There is Detect Ready to Detect washer to user is loadedrunning Washing problem a running Unload. unloaded Load loading and withor error. problem complete Clean. (e.g., Dirty machine ready to acceler-or error. with door run. ometer Stop the acceler- open for and/ormachine ometer greater temperat immedi- and/or than ure. ately. temper-threshold Detect Alert ature. time, rotation appro- Detect etc.). ofpriate rotation Detect device people. of rotation around device ofdevice specified around around axis. specified specified axis. axis.Coffee Machine Detected Machine Detected Machine Detected There isDetected Machine Detected Maker is user is loaded machine is loadedproblem a that the has user (Single empty loading and running and orerror problem machine completed unloading Serving) and not machine readyto running or error has running machine in use. Coffee run. Stop thecompleted and by cup machine its coffee taking installed immedi- runningcup is coffee under ately state ready. cup. dis- Alert without penser.appro- error. priate people. Coffee Machine Detected Machine DetectedMachine Detected There is Detected Machine Detected Maker is user isloaded machine is loaded problem a that the has empty (Coffee emptyloading and running and or error problem machine com- coffee Pot) andnot machine ready to running or error has pleted pot, or in use. Coffeerun. Stop the com- running cold pot machine pleted its and coffeeinstalled immedi- running coffee pot. under ately state pot is dis-Alert without ready to penser. appro- error. use. priate people. BlenderMachine Detected Machine Detected Machine Detected There is DetectedMachine Detected is user is loaded machine is loaded problem a that thehas empty empty loading and running and or error problem machine com-clean and not machine ready to running or error has pleted pitcher. inuse. Pitcher run. Stop the com- running. with machine pleted itsCleaning contents immedi- running required. installed. ately. stateAlert without appro- error. priate people. Micro- Machine DetectedMachine Detected Machine Detected There is Detected Machine Detectedwave is user is loaded machine is loaded problem a that the has userempty loading and running and or error problem machine com- unload- andnot machine ready to running or error has pleted ing in use. Detectedrun. Stop the com- running. machine. door machine pleted its closedimmedi- running ately. state Alert without appro- error. priate people.

TABLE 7.1 Cyclic Machine Specific Machine [Idle] → Working [Working] →Alert [Alert] Generic/ Waiting for Detected Machine is Determines anissue while Alert appropriate Other next cycle. working. performingperforming work or while idle person or people. work. (prolonged statein either part of cycle). Cashier Waiting for Detected Serving Detectedprolonged idle state or Alert appropriate Employee next servingcustomer. prolonged interaction with person or people. customer.customer. customer.

TABLE 3.2 Generic Machine/Abnormal State Detector (Vision/Sound Based)Specific Machine [Normal] → Alert [Alert] → Normal Generic/Other Normal.Problem detected Alert appropriate Problem resolved, or person orpeople. alert dismissed Water Heater Normal with no water on Problemdetected. Alert appropriate Normal situation ground. If Flir or IRperson of nature of detected with water Thermometer sensor problem.heater. available, temperature within expected range. Miter Saw/ Nodangerous situation Dangerous situation Use an actuator to Normalsituation Table Saw detected (e.g., no detected. immediately shut ofdetected. human body parts in the saw. Alert path of blade) appropriateperson or people. Roof of House No moisture or problem Moisture detectedNotify user. No moisture detected. Toilet (e.g., may Normal operation.Abnormal operation Alert appropriate Toilet normal be sound-based,detected (e.g., person or people. operation detected. proximity sensor,constantly running and/or vibration- water noise, water based only)detected on floor, etc.) Person/Baby Asleep or not crying. Detectedawake or Alert appropriate Detected asleep or not (e.g., Awake, crying.person or people. crying. Crying, etc.) Person Not pulling hair.Detected hand pulling Alert appropriate Detected hand moving (e.g.,monitor Not exhibiting motion hair, eyebrows, etc. person or people.away from hair, trichotillomania associated with disorder. Detectedmotion eyebrows, etc. (hair pulling) or associated with Detected motionother disorder.) disorder. associated with disorder has stopped. PersonPerson standing, sitting, Detected person Alert appropriate Detectedsituation (e.g., monitor laying down on bed. falling/or laying person onpeople. resolved. not fallen). Person not laying down hunched over onfloor. Optionally, alert or hunched over on floor. emergency services.

TABLE 4.2 Neglectable Machine Specific Machine [Normal] → Vulnerable[Vulnerable] → Alert [Alert] → Normal Generic/ Normal. Detected the Themachine Determines that the Alert Detected machine Other machine has isin a machine has been appropriate placed in normal been placedvulnerable neglected while in the person or state, or alert in a state.vulnerable state people. dismissed vulnerable state. Refrigerator Not inuse Detected Door open Detected prolonged door Alert Detected door withdoor door open. while in use. open state while not near appropriateclosed. closed. a human. Detected person or warning beeping from people.refrigerator for prolonged time without user interaction. Stove Not inuse Detected Heating or Detected prolonged Alert Detected cooling. withdoor heating or hot, vulnerable while not appropriate closed. hot.Heating or near a human Detected person or hot with door timer going offfor people. open. prolonged time without user interaction. Oven Not inuse Detected Steady state Detected steady state Alert Detected oven offwith door heating or heating or heating or hot for appropriate orcooling and closed. hot. hot. prolonged period. person or door closed.Detected timer on oven people. beeping for prolonged time without userinteraction. Sink Not in Detected Running. Detected prolonged on AlertDetected water use/off. running state while not near a appropriateturned off, human person or people. Television Not in Detected on. On.Detected prolonged on Emit infrared Detected machine use/off state whilenot near a signal to turn placed in normal human off the state, or alerttelevision. dismissed Alert appropriate person or people. Curling Not inDetected Steady state Detected prolonged Remove power Detected machineIron use/off. heating or heating or steady state heating or from theplaced in normal hot. hot while in hot while not near a machine. state,or alert use by human Alert dismissed person. appropriate person orpeople. Garage Closed Detected Open. Detected prolonged Alert Detectedclosed/ Door open/ period of time open appropriate closing. opening.before or after human/ person or car activity. people. Escalator orOff/not Detected On/moving. Determine an issue or Stop motion Detected aperiod Moving moving people dangerous situation immediately. of timewith no Walkway approaching involving someone Alert people on the theentrance riding on the escalator or appropriate escalator or to thedevice, moving walkway. (e.g., person or moving walkway increasedcurrent to people. and nobody in the motor indicating jam, area of thedevice. visual indication of jam, screams or other loud noises detected,etc.)

TABLE 5.2 Single User Machine Specific Machine [Available] → In Use [InUse] → Alert [Alert] → Available Generic/ Machine is Detected thatMachine is A problem is detected Alert appropriate Detected that theOther available the machine occupied with the machine or it person orpeople, machine has has become is flagged as out of and/or note thatbecome occupied order. the machine is unoccupied. (If not available from[Alert], the because it is out alert flag is reset) of order. Room Roomis Detected Human Detected sign on the Alert appropriate Human observednot human observed to door, cleaning cart in person or people, leavingthe room. occupied entering the have gone in front, or flagged as and/ornote that (If from [Alert], room the room out of order the machine isthe alert flag is not available reset). because it is out of order.Parking Parking Detected car Car is A problem is detected Alertappropriate Car observed Spot spot is entering the occupying with theparking spot person or people, leaving the spot. empty parking spot orit is flagged as out and/or note that or occupying the parking of orderor reserved the machine is the parking spot. Optionally, a non car notavailable spot object is detected to because it is out be in the spacein the of order. space. Seat Seat is not Detected Object or A problem isdetected Alert appropriate Person observed occupied. i object or humanwith the parking spot person or people, leaving the seat. humanoccupying or it is flagged as out and/or note that occupying the theseat. of order or reserved the machine is seat. Optionally, detect notavailable that an object is in because it is out seat, alert if in seatof order. too long.

TABLE 6.2 Loadable Machine Specific Machine [Load] → Loaded [Loaded] →Running [Running] → Alert [Alert] → Unload Generic/ Machine DetectedMachine is Detected Machine is Detected There is a Detected Other isempty user loading loaded and machine loaded and problem problem or thatthe and not in machine ready to running running or error error machineuse. run. Stop the has machine completed immediately its running Alertstate appropriate without people. error. Washing Machine DetectedMachine is Detected Machine is Detected There is a Detected Machine isempty user loading loaded and machine loaded and i problem problem orthat the and not in machine ready to running running or error errormachine use. Detected run. Stop the has door closed machine completedimmediately its running Alert state appropriate without people. error.Dryer Machine Detected Machine is Detected Machine is Detected There isa Detected is empty user loading loaded and machine loaded and problemproblem or that the and not in machine ready to running running or errorerror machine use. Detected run. Stop the has door closed machinecompleted immediately its running Alert state appropriate withoutpeople. error. Dishwasher Machine Detected Machine is Detected Machineis Detected There is a Detected is empty user loading loaded and machineloaded and problem problem or that the and not in machine ready torunning running or error error machine use. Detected run. Stop the hasdoor closed machine completed immediately its running Alert stateappropriate without people. error. Coffee Machine Detected Machine isDetected Machine is Detected There is a Detected Maker is empty userloading loaded and machine loaded and problem problem or that the(Single and not in machine ready to running running or error error.machine Serving) use. Coffee cup run. Stop the has installed machinecompleted under immediately. its running dispenser. Alert stateappropriate without people. error. Coffee Machine Detected Machine isDetected Machine is Detected There is a Detected Maker is empty userloading loaded and machine loaded and problem problem or that the(Coffee and not in machine ready to running running or error error.machine Pot) use. Coffee pot run. Stop the has installed machinecompleted under immediately. its running dispenser. Alert stateappropriate without people. error. Blender Machine Detected Machine isDetected Machine is Detected There is a Detected is empty user loadingloaded and machine loaded and problem problem or that the and not inmachine ready to running running or error error machine use. Pitcherwith run. Stop the has contents ma chine completed installed.immediately. its running Alert state appropriate without people. error.Microwave Machine Detected Machine is Detected Machine is Detected Thereis a Detected is empty user loading loaded and machine loaded andproblem problem or that the and not in machine ready to running runningor error error machine use. Detected run. Stop the has door closedmachine completed immediately its running Alert state appropriatewithout people. error.

TABLE 7.2 Cyclic Machine Specific Machine [Idle] → Working [Working] →Alert [Alert] Generic/ Waiting for Detected Machine is Determines anissue while Alert appropriate Other next cycle. working. performingperforming work or person or people. work. while idle (prolonged statein either part of cycle). Cashier Waiting for Detected Serving Detectedprolonged Alert appropriate Employee next serving customer. idle stateor prolonged person or people. customer. customer. interaction withcustomer.

With reference to FIG. 7A, an illustration of an example accelerometer-or sensor-based observer 100 such as device 700 attached to an exampleappliance 702 (e.g., a dishwasher, etc.) is provided. Dishwasherappliance 702 has a door that opens toward the ground on a horizontalaxis. Observer 700 is magnetically (or otherwise) attached to the doorof the dishwasher appliance 702. Depending on the inferred state of thedishwasher appliance 702, the observer 700 updates its display (e.g., ane-Ink display, a remote server/mobile phone app, etc.) to indicate thestate of the dishwasher such as by stating “CLEAN”/“DONE” 712,“WASHING”/“RUNNING” 714, or “DIRTY”/“LOAD” 716.

With reference to FIG. 8B, a three-state state diagram of the exampledishwasher observer 700 depicted in FIG. 7A is provided. At 822, thestate is “Load.” If a sustained vibration above a threshold level isdetected while in the “Load” state, then the transition (1) 823 from the“Load” 822 to “Running” 824 state occurs. Optionally, the sustainedvibration considered for this transition may be limited to the vibrationin the Z axis only, or the vibration in the X-Y plane only. Thesustained vibration may be measured using the accelerometer and a FastFourier Transform as illustrated elsewhere herein; or any other methodmay be used. At 824, the state is “Running”. If the sustained vibrationabove a threshold level stops while in the “Running” state, a transition(2) 825 to the “Done” 826 state occurs. At 826, the state is “Done.” Ifa sustained vibration above a threshold level is detected while in the“Done” state, then the transition (1) 823 from the “Done” 826 to“Running” 824 state occurs. If the door of the dishwasher is opened pasta threshold angle (e.g., 45 degrees down) for a sufficient period oftime top indicate that a user has unloaded the clean dishes, then thetransition (4) 828 from the “Done” 826 state to the “Load” 822 stateoccurs. If in the “Done” 826 state or the “Load” 822 state, the observer100 detects that it has been flipped in the X-Y plane or that adesignated button has been pressed, the state is reversed from “Done”826 to “Load” 822 or vice versa via transition (3) 827.

With reference to FIG. 7B, an illustration of an exampleaccelerometer-based observer 100 such as either device 700 or 701attached to an example top-load appliance 704 (e.g., clothes washingmachine, etc.) is provided. Top load washing appliance 704 has a doorthat opens up a horizontal axis. Observer 100 is optionally magneticallyattached to the door of the top-load washing appliance such asillustrated by device 700, or may also be attached to the front of theappliance such as illustrated by device 701. Depending on the inferredstate of the appliance 704, the observer 100 updates its display (e.g.,an c-Ink display, a remote server/mobile phone app, etc.) to indicatethe state of the appliance such as by stating “CLEAN”/“DONE” 712,“WASHING”/“RUNNING” 714, or “DIRTY”/“LOAD” 716.

With reference to FIG. 8B, a three-state state diagram of the exampleappliance observer 700 depicted in FIG. 7B is provided. At 822, thestate is “Load.” If a sustained vibration above a threshold level isdetected while in the “Load” state, then the transition (1) 823 from the“Load” 822 to “Running” 824 state occurs. Optionally, the sustainedvibration considered for this transition may be limited to the vibrationin the Z axis only, or the vibration in the X-Y plane only). At 824, thestate is “Running”. If the sustained vibration above a threshold levelstops while in the “Running” 824 state, a transition (2) 825 to the“Done” 826 state occurs. At 826, the state is “Done.” If a sustainedvibration above a threshold level is detected while in the “Done” state,then the transition (1) 823 from the “Done” 826 to “Running” 824 stateoccurs. If the door of the appliance is opened past a threshold angle(e.g., 45 degrees up) for a sufficient period of time top indicate thata user has unloaded the wet clothes to be placed in the dryer, then thetransition (4) 828 from the “Done” state to the “Load” 822 state occurs.If in the “Done” 826 state or the “Load” 822 state, the observer 100detects that it has been flipped in the X-Z plane or that a designatedbutton has been pressed, the state is reversed from “Done” 826 to “Load”822 or vice versa via transition (3) 827.

With reference to FIG. 7C, an illustration of an exampleaccelerometer-based observer 100 such as device 700 or 701 attached toan example appliance 706 (e.g., front-load washing machine or a dryer,etc.) is provided. Top load washing appliance 706 has a door that opensoutward on a vertical axis. Observer 100 is optionally attached to thedoor of the front-load washing or dryer appliance as illustrated bydevice 700 or may also be attached to the top of the appliance asillustrated by device 701. Depending on the inferred state of theappliance 706, the observer 100 updates its display (e.g., an e-Inkdisplay, a remote server/mobile phone app, etc.) to indicate the stateof the appliance such as by stating “CLEAN”/“DONE” 712,“WASHING”/“RUNNING” 714, or “DIRTY”/“LOAD” 716.

With reference to FIG. 8B, a three-state state diagram of the exampleappliance observer 700 depicted in FIG. 7C is provided. At 822, thestate is “Load.” If a sustained vibration above a threshold level isdetected while in the “Load” 822 state, then the transition (1) 823 fromthe “Load” 822 to “Running” 824 state occurs. Optionally, the sustainedvibration considered for this transition may be limited to the vibrationin the Z axis only, or the vibration in the X-Y plane only). At 824, thestate is “Running.” If the sustained vibration above a threshold levelstops while in the “Running” 824 state, a transition (2) 825 to the“Done” 826 state occurs. At 826, the state is “Done.” If a sustainedvibration above a threshold level is detected while in the “Done” state,then the transition (1) 823 from the “Done” 826 to “Running” 824 stateoccurs. If the door of the appliance is opened past a threshold angle(e.g., 45 degrees outward based on accelerometer, magnetometer,gyroscope, IMU, etc.) for a sufficient period of time top indicate thata user has unloaded the clothes, then the transition (4) 828 from the“Done” 826 state to the “Load” 822 state occurs.

If in the “Done” 826 state or the “Load” 822 state, the observer 100detects that it has been flipped in the X-Y plane or that a designatedbutton has been pressed, the state is reversed from “Done” 826 to “Load”822 or vice versa via transition (3) 827.

Additional examples of three-state machine states and transitions fordifferent appliances are provided in Table 2.

With reference to FIG. 7D, an illustration of an exampleaccelerometer-based observer 100 attached to an example garage door 708is provided. Garage door 708 is a vertically oriented plane when closed720, and a horizontal oriented plane when open 722. Observer 100 isattached to the inside of the garage door with adhesive, tape, ormagnets as illustrated by device 700. Depending on the inferred state ofthe garage door 708, the observer 100 updates its display (e.g., ane-Ink display, a remote server/mobile phone app, etc.) to indicate thestate of the garage door 708 such as by stating “OPEN” 726 or “CLOSED”724.

With reference to FIG. 8A, a two-state state diagram of the exampleappliance observer 100 depicted in FIG. 7D is provided. At 812, thestate is “Open.” While in the “Open” 812 state, the gravity vector isdetected to be in the +Z axis. If the gravity vector is detected tochange to the −Y axis, a transition (1) 813 to the “Closed” 814 stateoccurs. At 814, the state is “Closed.” While in the “Closed” 814 state,if the gravity vector is detected to change to the +Z axis, then thetransition (2)815 from the “Closed” 814 to “Open” 812 state occurs. Ifin the “Closed” 814 state or the “Open” 812 state, the observer 100detects that it has been flipped in the X-Y plane or that a designatedbutton has been pressed, the state is reversed from “Open” 812 to“Closed” 814 or vice versa via transition (3)816.

With reference to FIG. 7E, an illustration of an example observer 700attached to an example refrigerator door 750 with either an object orrange detector, a gyroscope, and/or a magnetometer, is provided.Refrigerator door 750 is a vertically oriented plane when closed, andalso a vertically oriented plane when open. The door is rotatedapproximately 90 degrees in space. The observer can, for example, detectif the door is open or closed based on: (A) a change in or specificvalue(s)/range of the magnetic field using a magnetometer, (B) a changein or specific value(s)/range of angular rotation using a gyroscope, (C)a change in whether an adjacent object is detected using an infraredemitter and detector, or (D) a change in range from a LIDAR or distancemeasuring device, and/or (E) any combination of the above or othersensor readings. Depending on the inferred state of the refrigeratordoor 750, the observer 700 updates its E-Ink display to indicate thestate of the refrigerator door 750 such as by stating “OPEN” 754 or“CLOSED” 752. If the door is left open for a prolonged period of time,the observer 700 may state “ALERT” 756 and notify the appropriate peoplevia notification or other methods. Another example observer 757 maysimilarly be placed in or on a freezer drawer 751. Various statemachines may be used with a refrigerator, including for example thoseshown in at least FIGS. 8C, 9B, 9H.

With reference to FIG. 7F, an illustration of an example accelerometer-or sensor-based observer 100 such as device 700 attached to an exampleheating appliance 702 (e.g., an oven, etc.) is provided. Heatingappliance 760 has a door that opens toward the ground on a horizontalaxis. Observer 700 is magnetically (or otherwise) attached to the doorof the appliance 760. Depending on the inferred state of the heatingappliance 760, the observer 700 updates its display (e.g., an e-Inkdisplay, a remote server/mobile phone app, etc.) to indicate the stateof the dishwasher such as by stating “COOL” 762, or “HOT” 764. If thedoor is left open for a prolonged period of time or the oven is left onfor a prolonged period of time, the observer 700 may state “ALERT” 756and notify the appropriate people via notification or other methods.Various state machines may be used with a refrigerator, including forexample those shown in at least FIG. 8C.

Additional examples of two-state machine states and transitions fordifferent appliances are provided in Table 1.

With reference to FIG. 9A, an exemplary abnormal state detector (orgeneric machine) state diagram is depicted. An abnormal state detectormay be used with an observer 100 in conjunction with any person,machine, appliance, object, etc. At 900, the state is “Normal.” See alsoTable 3 [Normal] column. If an abnormal condition is detected while in a“Normal” state, the state is updated via transition 904 to the “Alert”state 902. See also Table 3 “-> Alert” column. At 902, the state is“Alert” and an appropriate action is taken. See also Table 3 “[Alert]”column. While in the “Alert” state, if a normal condition is detected(optionally for a threshold period of time), or if there is a resetinput to the observer, then the state is updated via transition 906 tothe “Normal” state. See also Table 3 “-> Normal” column. A reset inputmay also trigger transition 908 from the “Alert” state 902 to the“Normal” state 900. Additional examples of states and transitions forvarious observed items are provided in Tables 3.1 and 3.2.

With reference to FIG. 9B, an exemplary openable machine (or neglectablemachine) state diagram is depicted. An openable machine state machinemay be used with an observer 100 in conjunction with any person,machine, appliance, object, etc. but it is particularly suited formachines that are to be utilized when in the proximity of a person. At910, the state is “Closed.” See also Table 4 [Normal] column. If it isdetected that the machine is opened, the state is updated via transition916 to the “Open” state 912. For example, it is detected that a useropened a door to a refrigerator, garage, or other device. See also Table4 “-> Open” column. At 912, the state is “Open.” If a it is detectedthat the machine is closed, the state is updated via transition 918 tothe “Closed” state 910. See also Table 4 “-> Normal” column. If anabnormal, neglected, or alert condition is detected while in an “Open”state 912, the state is updated via transition 911 to the “Alert” state914. See also Table 4 “-> Alert” column. At 914, the state is “Alert.”In the alert state, the observer 100 causes an appropriate action to betaken. See also Table 4 “[Alert]” column. If it is detected that themachine has been placed back into a closed condition (e.g., the userclosing the door to the refrigerator, garage, or other device, etc.),the state is updated to the “Closed” state via transition 918.Additional examples of states and transitions for various observed itemsare provided in Tables 4.1 and 4.2.

With reference to FIG. 9C, an exemplary neglectable machine (or openablemachine) state diagram is depicted. A neglectable machine state machinemay be used with an observer 100 in conjunction with any person,machine, appliance, object, etc. but it is particularly suited formachines that are to be utilized when in the proximity of a person. At920, the state is “Normal.” See also Table 4 [Normal] column. If avulnerable condition (i.e., a condition that should only occur when themachine is supervised) is detected while in a “Normal” state 920, thestate is updated via transition 926 to the “Vulnerable” state 922. Forexample, it is detected that a user turned on a neglectable machine. Seealso Table 4 “-> Vulnerable” column. At 922, the state is “Vulnerable.”If a normal condition is detected when in the vulnerable state (e.g.,the user turning off the machine), the state is updated via transition928 to the “Normal” state 920. See also Table 4 “-> Normal” column. Ifan abnormal or neglected condition is detected while in a “Vulnerable”state 922, the state is updated via transition 921 to the “Alert” state924. See also Table 4 “-> Alert” column. At 924, the state is “Alert.”In the “Alert” state 924, the observer 100 causes an appropriate actionto be taken. See also Table 4 “[Alert]” column. If it is detected that auser is no longer neglecting the machine, the stat is updated to“Vulnerable” 922 via transition 926. If it is detected that the machinehas been placed back into a normal condition (e.g., the user turning offthe machine), the state is updated to the “Normal” state 920 viatransition 928. Additional examples of states and transitions forvarious observed items are provided in Tables 4.1 and 4.2.

With reference to FIG. 9D, an exemplary single-user machine statediagram is depicted. A single-user machine state machine may be usedwith an observer 100 in conjunction with any person, machine, appliance,object, etc. but it is particularly suited for machines that are to beutilized only by a single person or group of people such as a room,parking spot, or seat(s). At 930, the state is “Available.” See alsoTable 5 [Available] column. If it is detected that the machine hasbecome occupied while in an “Available” state 930, the state is updatedvia transition 936 to the “In Use” state 932. See also Table 5 “-> InUse” column. If an abnormal condition is detected or flagged while in an“Available” state 930, the state is updated via transition 931 to the“Alert” state 934. See also Table 5 “-> Alert” column. At 932, the stateis “In Use.” If it is detected that the machine has becomefree/unoccupied while in an “In Use” state 932, the state is updated viatransition 938 to the “Available” state 930. See also Table 5 “->Available” column. If an abnormal condition is detected or flagged whilein an “In Use” state 932, the state is updated via transition 931 to the“Alert” state 934. See also Table 5 “-> Alert” column. At 934, the stateis “Alert.” In the “Alert” state 934, the observer 100 causes anappropriate action to be taken. See also Table 5 “[Alert]” column. If itis detected that the machine has been placed back into a normalcondition or if the state is reset, the state is updated to the“Available” state 930 via transition 933. Additional examples of statesand transitions for various observed items are provided in Tables 5.1and 5.2.

With reference to FIG. 9E, an exemplary loadable machine state diagramis depicted. A loadable machine state machine may be used with anobserver 100 in conjunction with any person, machine, appliance, object,etc. but it is particularly suited for machines that are loaded, run andthen must be unloaded such as a dishwasher, washing machine, or dryer.At 940, the state is “Load” or “Load.” See also Table 6 [Load] column.If it is detected that the machine is being loaded by a user, the stateis updated via transition 941 to the “Loaded” state 942. See also Table6 “-> Loaded” column. At 942, the state is “Loaded.” If it is detectedthat the machine has started to run, the state is updated via transition943 to the “Running” state 944. See also Table 6 “-> Running” column. Ifit is detected that the machine is being unloaded by a user, the stateis updated via transition 949 to the “Load” state 940. See also Table 6“-> Ready to Load” column. At 944, the state is “Running.” If it isdetected that the machine has finished running, the state is updated viatransition 947 to the “Unload” state 946. See also Table 6 “-> Ready toUnload” column. If an abnormal condition is detected or flagged while inan “Running” state, the state is updated via transition 945 to the“Alert” state. See also Table 6 “->Alert” column. At 948, the state is“Error” or “Alert.” In the “Alert” state 948, the observer 100 causes anappropriate action to be taken. See also Table 6 “[Alert]” column. If itis detected that the machine has started to run, the state is updatedvia transition 943 to the “Running” state 944. See also Table 6 “->Running” column. If it is detected that the machine is being unloaded bya user, the state is updated via transition 949 to the “Load” state 940.See also Table 6 “-> Ready to Load” column. At 946, the state is “Done”or “Unload.” If it is detected that the machine has started to run, thestate is updated via transition 943 to the “Running” state. See alsoTable 6 “-> Running” column. If it is detected that the machine is beingunloaded by a user, the state is updated via transition 949 to the“Load” state 940. See also Table 6 “-> Ready to Load” column. Additionalexamples of states and transitions for various observed items areprovided in Tables 6.1 and 6.2.

With reference to FIG. 9F, an exemplary cyclic machine state diagram isdepicted. A cyclic machine state machine may be used with an observer100 in conjunction with any person, machine, appliance, object, etc. butit is particularly suited for machines that are to be monitored toensure that they are operating efficiently. At 950, the state is “Idle.”See also Table 7 [Idle] column. If it is detected that the machine hasbegun working, the state is updated via transition 956 to the “Working”state 952. At 952, the state is “Working.” See also Table 7 [Working]column. If it is detected that the machine has completed a task and isidle again, the state is updated via transition 958 to the “Idle” state.See also Table 7 “-> Idle” column. If an abnormal condition is detectedwhile in a “Working” state, the state is updated via transition 951 tothe “Alert” state 954. See also Table 7 “-> Alert” column. For example,a cycle is taking longer than normal or some other event occurs that isnot typical. At 954, the state is “Alert.” See also Table 7 [Alert]column. In the “Alert” state 954, the observer 100 causes an appropriateaction to be taken. See also Table 7 “[Alert]” column. If it is detectedthat the machine has begun working again, the state is updated viatransition 956 to the “Working” state 952. If it is detected that themachine has completed a task and is idle again, the state is updated viatransition 958 to the “Idle” state 950. See also Table 7 “-> Idle”column. Additional examples of states and transitions for variousobserved items are provided in Tables 7.1 and 7.2.

There are several examples of applications of the above types of statemachines discussed in Tables 3-7. For example, at least the followingdifferent types of machines, objects, and people can be observed in ahome or office environment:

-   -   Mailbox (e.g., Loadable Machine)    -   Garage Door (e.g., Neglectable Machine)    -   Oven (e.g., Neglectable Machine)        -   May use IR or visual light        -   Example States:            -   Off            -   On/Preheating            -   On/Ready to Use            -   On/Cooking            -   On/Empty            -   On/Alert (user forgot something was cooking, left on                while empty)    -   Stove (e.g., Neglectable Machine)        -   May use IR or visual light        -   Each individual burner may be monitored, or the whole stove            can be treated as one appliance.        -   Example States:            -   Off            -   On/Preheating            -   On/Ready to Use            -   On/Cooking            -   On/Empty            -   On/Alert (user forgot something was cooking, left on                while empty)    -   Refrigerator (e.g., Neglectable Machine)        -   Example States:            -   Closed            -   Open/Loading            -   Open/Alert (open and user forgot to close for an                extended period of time)    -   Sink (e.g., Neglectable Machine)    -   Coffee Maker (e.g., Loadable Machine)    -   Blender (e.g., Loadable Machine)    -   Microwave (e.g., Loadable Machine)    -   Dish Washer (e.g., Loadable Machine)    -   Washing Machine (e.g., Loadable Machine)    -   Dryer (e.g., Loadable Machine)    -   Hot Water Heater (e.g., Generic Machine)    -   Sink (e.g., Neglectable Machine)    -   Curling Iron (e.g., Neglectable Machine)        -   When neglected, alert and turn off power if the observer is            interfaced with an actuator for the curling iron.    -   Bathroom Generally (e.g., Single User Machine)—To maintain        privacy, monitors door—user entering and exiting. However, in        instances where privacy is not a concern could monitor        individual fixtures in the bathroom.        -   Toilet (e.g., Single User Machine)        -   Shower (e.g., Single User Machine)    -   TV (e.g., Neglectable Machine)        -   Example States            -   Off            -   On/Someone Watching            -   On/Nobody Watching    -   Lights (e.g., Neglectable Machine)        -   Example States            -   Off            -   On/People There            -   On/People Away    -   Saw (e.g., Generic Machine)    -   Drill press (e.g., Generic Machine)    -   Other

In addition, in a retail or industrial application, the followingobjects, machines and people may be observed:

-   -   Queue management (e.g., Single User Machine, Cyclic Machine).        -   Detect that a register is available/cashier is idle.        -   Use image processing on front of queue to vocalize            description of next customer and instruction to go to the            available register.            -   “Man in green shirt, please proceed to register number                5.”    -   Employee metrics        -   Register cashier state monitoring (e.g., Cyclic Machine)            -   Example States:                -   Ready for Customer                -   Serving Customer                -   Abnormal State (taking too long, customer appearing                    to have issue, etc.)    -   Security Systems        -   Recognizing “stealing” or opening product in store. (e.g.,            Abnormal State Detector).    -   Escalator (e.g., Neglectable Machine)    -   Parking Spaces (e.g., Single User Machine)    -   Seating (e.g., Single User Machine)    -   Dispenser (e.g., Generic Machine—Abnormal when out of soap.)        -   Paper Towel Dispenser (e.g., Generic Machine—Abnormal when            out of soap.)        -   Soap Dispenser (e.g., Generic Machine—Abnormal when out of            soap.)

In addition to the above, more complex state machines and combinationsof state machines may be created to represent and track observedobjects.

II.2 Hierarchical (Nested) State Machines. Hierarchical or nested statemachines may also be implemented.

For purposes of illustration and without limitation, with reference toFIG. 9G, an observed dishwasher may be associated with a hierarchal(nested) state machine 967. The dishwasher may be associated with acyclic machine (or similar) state machine, and the “load” state 963 mayfurther have a two-state (or similar) state machine that tracks whetherthe dishwasher door is open 960 or closed 962 nested within. Thetransition 961 from the “Load” state 963 to the “Running” state 964 mayalso only be allowed to trigger when the two-state machine for the“Load” state 963 is in the “Closed” state 962.

II.3 Parallel State Machines. Parallel state machines may beimplemented. For example, a single observer may track multiple observedobjects in parallel, or multiple different states associated with asingle observed object.

For purposes of illustration and without limitation, with reference toFIG. 9H, an observed refrigerator 970 may be associated with multipleparallel state machines such as: (1) Refrigerator Temperature: AbnormalState Detector 972, (2) Refrigerator Door: Neglectable Machine 976, (3)Freezer Temperature: Abnormal State Detector 974 and/or (4) FreezerDoor: Neglectable Machine 978. The refrigerator and freezer temperatureabnormal state detectors may be configured to monitor the temperature ofthe respective portions of the appliance using any methods discussedherein or otherwise apparent. For example, sensors in the freezer andrefrigerator portions of the appliance, sensors mounted on the exteriorof the freezer and/or refrigerator doors of the appliance, a thermalcamera trained on the appliance with knowledge of what portion of thethermal image corresponds to each compartment of the appliance, etc. maybe used in conjunction with the freezer and refrigerator abnormal statedetectors. Similarly, the state of the doors may be tracked using motionsensors or video and used in conjunction with the freezer andrefrigerator door neglectable machines.

As another example, a stacked washer and dryer may be observed in asimilar manner utilizing two cyclic machine state machine models (orsimilar).

II.4 History States. A memory may also be associated with the statemachine in the form of, for example, a history state. The history statemay be utilized to story any type of information.

For example, for purposes of illustration and without limitation, ahistory state that keeps a count of the number of times a machine hasrun may be utilized to determine when new consumables (e.g., dishwashertablets, etc.) should be ordered. Upon reaching a certain count, thishistory state may trigger an alert to be sent. If it is determined, viaa linked purchase history or API, that new consumables have beenobtained, the history state may be reset. Alternatively, the historystate may be reset after the notification is sent.

II.5 Situational State Machines.

In various forms of the invention, different state machines may beimplemented based on varying conditions. For example, a state machinemay always be active; a state machine may only be active if a conditionis met (e.g., X=Y, etc.); and/or a state machine may only be active ifanother state machine state is in a specific state.

For purposes of illustration and without limitation, if a user isstopped at a stop light an observer in an automobile may observe thestop lights. A state machine associated with a stop light may beactivated to determine when to start an internal combustion engine orget the electric motor ready for a green light.

As another example, in forms of the invention implementing a pan/tiltcamera sensor as part of an observer, the observer may keep track ofregions to which to apply rules and state machines to based onorientation of camera view. A pan/tilt camera (or any other cameracapable of having different orientations or views) may be configured toobserve different objects at different azimuth and elevation valuesrelative to the camera. Based on the object in view of the camera (e.g.,as determined by the current approximate camera azimuth and elevation,image recognition, etc.) the system may apply a different state machineto the images received from the camera.

For example, at azimuth 0 elevation 0, the pan/tilt camera observer 100may observe a refrigerator. At azimuth 90 elevation 0 the pan/tiltcamera may observe an oven. At azimuth 90 elevation 45, the pan/tiltcamera may observe a kitchen light fixture. The pan/tilt camera observermay vary its view by changing azimuth and elevation as to scan the room.When approximately at the azimuth and elevation values for the variousmachines and appliances, it may supply the images and sound received viathe camera to a state machine assigned to the appliance within thepan/tilt camera's view. For example, when at azimuth 90 and elevation 0,the oven state machine (e.g., a neglectable machine, etc.) state machinemay be consulted and/or updated based on the new images and videoreceived by the pan/tilt camera.

In various forms of the invention, state machines may be activated anddeactivated based on a location-based state machine lookup. For example,while out in the world a mobile device may determine its location andlookup state machines that are associated with nearby physical observedobjects. The state machine may be located based on any one orcombination of:

-   -   location    -   altitude    -   image recognition of an imaged object    -   category    -   selection from list    -   ownership and permissions    -   etc.

For example, a user may want to know about a state machine associatedwith a particular hospital room door. The location of the phone in thehospital may be determined based on latitude and longitude, a spatialdepth map, Wi-Fi location, or other means. The state machine server maybe queried with this information to return the status of the hospitalroom(s) near the user.

Other examples include:

-   -   State machine tracks if a nurse has been into the room in the        last hour, if not an alert is triggered (neglectable machine);    -   Can pull up a list based on location (and optionally altitude)        to get the state machine associated with nearby machines, and        query the status of the machine;

In some forms of the invention state machines and observer devices canbe grouped into projects. These projects' can be tagged with location orother information so that they can be accessed by third-party mobiledevices that can look up these state machines/projects. Projects mayhave observers associated with them and the mobile device may simplyrequest state information from, or otherwise interacts or manages, theproject/state machines.

In some forms of the invention, mobile devices may also be able to turninto an observer for the state machine based on a location-based lookup.

-   -   Example, different autonomous cars being an observer for a        location-based state machine at different times.    -   Example, different drones being an observer for a location-based        state machine at different times.

II.6 All Possible State Machine Configurations. Any possiblecombination, permutation, nesting, parallel or other form of statemachines may be crafted and associated with any type of observer and forany observed object. See Source Tables 8 and 9.

III. Classifiers.

State machine states and/or transitions may be determined or triggeredwith a classifier that acts on data that is collected from the observer100. Classifiers take information and assign a classification to theinformation based on shared characteristics or features of theinformation. The information, once classified, can be the basis forcausing a transition from one state to another state of a state machine.

III.1 Non-Neural Network Classifiers

In many instances, programming logic can be used to classify informationin order to determine if a transition from one state to another stateshould occur. For example, sensor data can be read and compared tothreshold values. As another example, functions can be performed on acollection of sensor data (e.g. a Fast Fourier Transform on a timeseries of sensor data, etc.) and the result, or several results, may beused in order to classify the information or activity.

For purposes of illustration and without limitation, with reference toFIG. 5A, some example sensor- or accelerometer-based observer devicedata for a loadable machine (e.g., dishwasher, washer, dryer, etc.) isshown. With respect to graph 502, a |Vibration| value is graphed overtime. The |Vibration| value may be based on at least one axis of anaccelerometer and a signal processing algorithm such as a Fast FourierTransform. With respect to graph 504, a |Temp.| value is graphed overtime. The |Temp.| value may be based on at least a temperature sensorand optionally an algorithm to compute a temperature value or smooth atemperature reading value over time. With respect to graph 506, a |DeltaGyro/Mag.| value is graphed over time. This value may be based on atleast one axis of a gyroscope and/or magnetometer and optionally analgorithm to compute a temperature value or smooth a temperature readingvalue over time.

With respect to the example data graphs 502, 504, and 506, time period508 represents a period of time where the door of the loadable machineis observed to be in an open orientation as indicated by “DeviceOrientation 2”, the relatively low value of |Vibration|, the relativelylow value for |Temp.|, and/or the spike in |Delta Gyro/Mag| at thebeginning and end of the period. From this data, the observer mayconclude that the observed appliance is “Ready to Load” or that the dooris open.

With respect to the example data graphs 502, 504, and 506, time period510 represents a period of time where the loadable machine is observedto be in a running state indicated by “Device Orientation 1”, theincreased value of |Vibration|, the increased value for |Temp.|, and/orthe spike in |Delta Gyro/Mag| at the beginning of the period. From thisdata, the observer may conclude that the observed appliance is“Running.”

With respect to the example data graphs 502, 504, and 506, time period512 represents a period of time where the loadable machine is observedto be done running and in a “clean” or “ready to unload” state indicatedby “Device Orientation 1”, the decreased value of |Vibration|, thedecreased value for |Temp.|, and/or the spike in |Delta Gyro/Mag| at theend of the period. From this data, the observer may conclude that theobserved appliance is “Done.”

With respect to the example data graphs 502, 504, and 506, time period514 represents a period of time where the door of the loadable machineis observed to be in an open orientation as indicated by “DeviceOrientation 2”, the relatively low value of |Vibration|, the relativelylow value for |Temp.|, and/or the spike in |Delta Gyro/Mag| at thebeginning of the period. From this data, the observer may conclude thatthe observed appliance is “Ready to Unoad” or that the door is open.

With respect to FIG. 5B, experimental Fast Fourier Transform datacollected using an accelerometer for five dishwasher washing cycles andthree dishwashers off cycles is shown. The columns represent differentdata sets, and the rows represent the output of the Fast FourierTransform. The “Average” row at the bottom is the average of the valuesproduced by the Fast Fourier Transform. Values obtained for a dishwasherin the washing state are shown in Cols B, C, D, H, and I. Valuesobtained for a dishwasher in the off state are shown in Cols E, F, andG. In this particular example, the “Average” values for the “washing”states are about: 197, 147, 166, 74, and 201; the “Average” values forthe “off” states are about: 28, 34, and 31. While these values will bedifferent for different variations of the observer hardware and choicesmade in implementing the FFT algorithm, the FFT Average for off cyclesis typically much lower than for washing cycles. From this information,threshold values for classifying whether a machine is “Running” or “NotRunning” can be determined for a particular implementation or form ofthe observer device. FIG. 5C is the FIG. 5B data in a line chart format.This method can be implemented on any hardware with accelerometer andprocessor such as ESP32, RPi, etc.

For purposes of illustration and without limitation, with reference toFIG. 6A, a flow chart for a process for taking sensor measurements,processing the sensor measurements, and determining whether a statechange should occur is depicted.

At act 620, a series of numerous measurements are taken over a shortperiod of time. For example, perhaps 100 measurements of the Z axis ofthe accelerometer over the course of a second are taken and recorded.This information is then processed using a signal processing algorithmsuch as a Fast Fourier Transform (FFT). A result, such as the averagevalue of the FFT output, may be obtained.

At act 622, the result is compared to a threshold value. If the resultis less than the threshold value, the flowchart proceeds to act 624. Ifthe result is greater than the threshold value, the flowchart proceedsto act 626. If the result is equal to the threshold value, the systemcan proceed to one or the other state depending on the particularimplementation.

At act 624, it is determined that it is unlikely that the current stateis X (e.g., “running,” etc.) or likely that the current state is Y(e.g., “done,” etc.) after comparing the result to the threshold. At act626, it is determined that it is likely that the current state is X(e.g., “running,” etc.) or unlikely that the current state is Y (e.g.,“done,” etc.) after comparing the result to the threshold. The specificstate determination may be dependent on the type of machine beingobserved and the current state of the machine. At act 628, theobservation is added to a measurement history. At act 630, if the past Mresults (e.g., 2 results) indicate that a state transition should occur,then the state machine model is instructed to move to the appropriatestate. At act 632, a delay is implemented, the microcontroller isinstructed to go to deep sleep (e.g., to conserve battery power, etc.)and wake up after a period of time (e.g., 1 minute, etc.) beforeproceeding back to act 620. For purposes of further illustration andwithout limitation, at least one if not more possible implementations ofa process as outlined above in FIG. 6A is provided as Source Tables 1,2, 3, 4, 5, 6, 9, and 10.

III.2 Neural Network Classification

In various forms of the invention, neural networks can be used asclassifiers in order to detect states, or state transition triggerevents. The neural networks can be used on their own, or in conjunctionwith other programming techniques such as the above described signalprocessing techniques or algorithms, and/or finite state machines inorder to track the state of an observed machine, and trigger statetransitions between tracked states.

For purposes of illustration and without limitation, with reference toFIG. 6B, input data 604 is fed into neural network classifier 602, and alikely state 606 probability is output. This output value can be used onits own to determine a state or trigger event, or the output value canbe used as the “result” (or the basis of a value that is turned into a“result”, etc.) that is fed into the act 622 comparison of FIG. 6A.Neural networks can be trained to classify information, with numerousdifferent features, based on examples and tags. Data, such as the dataillustrated in FIGS. 5A-C, may be utilized to train the neural network602.

For purposes of further illustration and without limitation, at leastone if not more possible implementations of a process as outlined aboveto train a neural network based on sensor data for a microcontroller isprovided as Source Tables 11, 12 and 13.

Alternatively, other data types such as image data, depth information,point cloud information, thermal imaging information, sound data,accelerometer data, other sensor data, video data, or any other orcombination of data may be utilized to train a neural networkclassifier.

For example, various types of neural networks may be implemented indifferent forms of the invention. Depending on the type of data to beprocessed, different neural network designs, data sets, or pre-trainedmodels may be used to construct a neural network from the followingmodels and datasets, or the like:

-   -   Image Data        -   Convolutional Neural Networks (CNNs)        -   ImageNet        -   MobileNet        -   Inception    -   Red Green Blue Depth (RGB-D) Image Data        -   RGB-D (Kinect) Object Dataset    -   PointCloud Data        -   PointNet    -   Thermal Image Data        -   ThermalNet    -   Sound Data        -   Conversion of Sound Data to a Spectrogram and then            processing with an imaged-based neural network.    -   Accelerometer Data    -   Video Data/Human Activity        -   ResNet        -   UCF101—Action Recognition Data Set        -   Continuous Video Classification    -   Other Datasets    -   Combinations of Above and/or Other Data Types

Any of the neural network types can be utilized independently or inconjunction with one another; for example, as part of a multi-pathneural network. For purposes of illustration and without limitation,with reference to FIG. 10, one possible example of a CNN LSTM Multi-PathNeural Network that may be used in various forms of the invention isdepicted. Any other neural network design or topology may be utilizedwith the invention.

The neural network may be trained from scratch, or transfer learningtechniques can be used to refine a pre-trained neural network. Forpurposes of illustration and without limitation, with reference to FIG.11, one possible process used in various forms of the invention forobtaining training data, training the neural network, operating theneural network and refining the neural network over time is depicted. Atact 1100, the camera or observer device is configured. At act 1101,training data is collected; it may be downloaded or captured fromsensors and/or imaging devices. At act 1102, data is categorized and/ortagged. At act 1103, a neural network machine learning model is trained(optionally via transfer learning). At act 1104, the neural networkmachine learning model is loaded into an observer device. At act 1105,the model is used with new input information. At act 1106, input datathat produces an incorrect action or output can be re-tagged or flagged.Then, using the corrected information, the process may re-start at act1103 to improve the deep neural network model.

III.2.A Transfer Learning.

It can be expensive and time consuming to train a neural network fromscratch. Transfer learning allows pre-trained deep neural networks to bere-trained using significantly less data then would be needed to trainthe deep neural network from scratch. With transfer learning, the“knowledge” that is already contained within the deep neural network istransferred from the prior learning task to the current task. The twotasks are not disjoint, and as such whatever network parameters thatmodel has learned through its prior extensive training may be leveragedwithout having to do expend time and resources performing that trainingagain.

In various forms of the invention, pre-trained image recognition neuralnetworks (e.g., ImageNet, etc.), or other types of neural networks, areused as the basis for a neural network and are re-trained via transferlearning and data collected by the observer device and tagged. This typeof process may be used to economically crate a custom classifier that isable to be used as the basis to determine the state of a machine model,and/or a transition trigger event.

For purposes of further illustration and without limitation, at leastone if not more possible implementations of a process to performtransfer learning on a neural network based on image data for a singleboard computer is provided as Source Tables 14, 15 and 16.

III.2.A.i A User Friendly Interface for Transfer Learning

In some forms of the invention, an interface (e.g., website, app, etc.)is provided that allows a user to upload data to tagged buckets ortagged data (e.g., text, picture, video, sound, point cloud data, acombination of different data types, other data types, etc.), and selectan appropriate pre-trained classifier as a basis to perform transferlearning on to add the desired classifications. This can be used togenerate a custom classifier economically and in a end-user-friendlymanner. The available classifiers to train may be filtered based on thetype of tagged data supplied by the user. The interface may be awebsite, application (desktop or mobile device). The interface may askfor the data and an associated tag (e.g., “loading appliance”,“appliance running”, “appliance error”, “appliance done”, “unloadingappliance”) or it may provide different “buckets” in which the data canbe uploaded via drag-and-drop (e.g., drag images for classification 1 toarea 1, for classification 2 to area 2, etc.). The user may be asked toprovide state names for the different classifications. The website orapplication may provide a model file that the user can download, thatthe user can send to a specific observer (or group of observers) via areference code or user ID and device ID, or an API endpoint that can beused to submit information and receive a classification or trigger eventdetermination.

III.2.B Training a Sensor-Based Neural Network Classifier. For aparticular machine condition that one desires to classify (e.g.,“machine running normally,” “machine running irregularly,” “machineoff,” “machine door open,” any state listed in any table or paragraphlisted herein, etc.), data can be collected and tagged. This data canthen be used to train or refine a neural network classifier.

1. Training Data Collection. For purposes of illustration and withoutlimitation, a microcontroller can be put into a data collection andtagging mode. Training data may be collected from a 3-axisaccelerometer, 3-axis gyroscope, 3-axis magnetometer, or any combinationthereof. The sensors may be sampled for a period of time (e.g., 1second, etc.) to obtain numerous samples. The collected data can be(pre-)processed before further use to train a neural network. Forexample, a fast Fourier transform may be performed on any of the abovedata. As one example for purposes of illustration and withoutlimitation, a fast Fourier transform may be performed on one axis of the3-axis accelerometer. The total of the FFT components, the average ofthe FFT components, the maximum FFT component and the standard deviationof the FFT components may be obtained. Any or all of the above data maybe recorded and tagged to associate the data with the known state of theappliance (e.g., running, idle, unloading, etc.) when the data wascollected. This data collection and tagging process may be repeated anynumber of time for the same state (e.g., running, done, loading, etc.)in the same or different conditions, or may be repeated for differentstates under varying conditions. Source Table 11 provides example Csource code that will produce a CSV file of training data from anaccelerometer on an Arduino compatible microcontroller. The CSV file canbe saved with a file name representing the state of the appliance duringthe data collection process (e.g., “running.csv,” “done.csv,” etc.)

2. Training a Neural Network. The tagged data CSV is uploaded to acomputer or network with sufficient processing power to train the modelor may be trained on the observer 100 device if sufficient computingpower is available. Source Table 12 for a Google Colab Python notebookthat will take the training data discussed above and produce a neuralnetwork classifier C header file for use on the Arduino compatiblemicrocontroller. Once the model is trained, then the trained neuralnetwork is then downloaded from the remote computer and may be used on amicrocontroller.

3. Using the Trained Neural Network Model on a Microcontroller. Thedownloaded neural network is loaded on the microcontroller, and then maybe used to classify the sensor data for use with the state machinemodels. Source Table 13 provides example C source code that willclassify newly observed accelerometer data into a state such as“running,” “done,” “unloading,” etc. based on the neural network createdfrom the training data.

4. Re-Training. To the extent that additional data is collected, theneural network model from step 2. Above can be improved by re-trainingwith additional data, or transfer learning with the additional collectedand tagged data.

5. Sharing Trained Models. For “internet of things” devices, thistraining data may be uploaded to a central repository of taggedinformation, so that all users of similar devices may benefit from thedata as it is incorporated into trained models for similar devices.

III.2.C Training an Imaged-Based Neural Network Classifier with MobileImageNet and Transfer Learning. Similarly to the above sensor-basedexample, for a particular machine condition that one desires to classify(e.g., “machine running normally,” “machine running irregularly,”“machine off,” “machine door open,” any state listed in any table orparagraph listed herein, etc.), image data can be collected and tagged.This data can then be used to train or refine a neural networkclassifier.

1. Training Data Collection. For purposes of illustration and withoutlimitation, a microcontroller or single board computer observer devicecan be put into a data collection and tagging mode. Training data may becollected from an imaging device such as a camera, other sensors orimaging devices, or any combination thereof. The camera may be used totake numerous pictures of the observed object for a known state invarying conditions. This data collection and tagging process may berepeated any number of times for the same state (e.g., running, done,loading, etc.) in the same or different conditions, or may be repeatedfor different states under varying conditions. Source Table 14 providesexample Python source code that will use a camera to take pictures inorder to on a Raspberry Pi single board computer. Similarly, SourceTable 6 provides depth camera image capture code. The image files can besaved in a folder with a name representing the state of the observedobject when the picture was taken (e.g., “./running/,” “./done/,”“./loading/,” etc.)

2. Training a Neural Network. The saved tagged image data is used trainthe observer 100 device using its local processor, or optionally aremote server. Source Table 14 provides example Python source code thatwill perform transfer learning on a pre-trained Mobile ImageNet neuralnetwork in order to add new classifications based on the saved taggedimages. Once the model is re-trained, the neural network can be used toclassify new images taken by the camera on the observer device.

3. Using the Trained Neural Network Model on a Microcontroller. Theneural network can be used on the single board computer observer, andthen may be used to classify the new image data for use with the statemachine models. Source Table 15 provides example Python source code willperform inference using the re-trained neural network to classify newlyobserved image data into an image category etc. based on the neuralnetwork created from the training data.

4. Re-Training. To the extent that additional data is collected, theneural network model from step 2. Above can be improved by re-trainingwith additional data, or transfer learning with the additional collectedand tagged data.

5. Sharing Trained Models. For “internet of things” devices, thistraining data may be uploaded to a central repository of taggedinformation, so that all users of similar devices may benefit from thedata as it is incorporated into trained models for similar devices.

IV. User Applications

Once the state is observed by an observer 100 device, the state iscommunicated to either a mobile device or an observer backend. In eitherevent, the state information can be made available to a user's or theirfamily members other devices in many ways.

For example, the state determination for an observed device may betransmitted to a television, set-top box, smart watch, mobile device,computer, or any other screen-based device. The observed device stateinformation can be presented as a notification or provided in anotification panel area or other easily accessible menu or widget thatis accessible to the user.

For purposes of illustration and without limitation, observed devicestate may be provided via an device (e.g., Apple TV, iPhone, AppleWatch, FireTV, etc.) widget that lists devices and current statusinformation, or may be presented as a toast-style (or any other style,etc.) notification message that appears momentarily on the screen aroundthe time the state of the machine changes or if there is a reminderevent about a “stale” state that requires attention (e.g., “your clotheswasher is done don't let your wet clothes continue to sit”, etc.).

As another example, the state determination for an observed device maybe provided via an application or website with observation and/orcontrol panel. Some example applications and/or website user interfacesare shown in FIGS. 12B-12J and Source Tables 17 and 18, discussedfurther below.

As another example, the state determination for an observed device maybe provided via an application programming interface (API) such as“RESTFUL” API that allows third party services to easily query thestatus or state of a device. Alternatively, new input information may besubmitted via an API and an updated state may be determined and returnedobviating the need to have the neural network model present on the edgeobserver device.

As another example, the state determination for an observed device maybe provided via a voice assistant notification or user query andresponse. An example voice notification and voice skill to retrieveobserved device information illustrated in FIGS. 13A-13I and SourceTables 19 and 20, discussed further below.

IV.A Mobile Device Applications and Notifications

In various forms of the invention, the observer 100 devices can becoupled to other systems that allow users to query and observe thecurrent state of the appliances observed by the observer. Any of thenotifications can be used in any of the different contexts (app, pushnotification, voice assistant query, voice assistant notification,etc.). The below examples are provided for illustration withoutlimitation as to the platform used.

For purposes of illustration and without limitation, with reference toFIG. 12A, the observer 100 experiences a state change at act 1202. Atact 1204, observer 100 causes its display to update (if applicable) andtransmits updated state information via a radio interface (e.g., BLEBeacon/Advertisement, Bluetooth communication session, Wi-Fi, cellular,etc.). At act 1206, the transmission causes either a notification orannouncement to be sent to a mobile device, email, or messagingplatform, or voice assistant backend.

Mobile Device Applications. In various forms of the invention, mobileapplications are provided that allow users to retrieve the current stateof observed devices and/or manage observer devices. Generally, theapplications may provide the following features: add observer devices,configure observer devices, and view and/or set the current status ofobserver devices/observed machines.

For purposes of illustration and without limitation, with reference toFIG. 12B, a main screen of a Bluetooth Low Energy (BLE) connectedobserver mobile application is shown. However, the mobile applicationmay also be used with other forms of the devices including Wi-Fi,cellular, or other connected observer devices. The main screen shows alisting of the observer devices such as an address or identifier of eachdevice (e.g., CCD31234, 15233293, AB325532, FA32FE2C, etc.) along withan indication of the type of machine or a friendly name that is observed(e.g., Dishwasher, Washing Machine, Dryer, Garage Door, etc.) and thestatus of each known device (e.g., Clean, Washing, Ready to Load,Closed, etc.). Color icons may also be used to show the state (e.g.,Green is Clean or Closed, Blue is Washing, Orange is Ready to Load,etc.). Furthermore, a battery icon can be applied to the information inorder to indicate a low battery that needs to be replaced or a lastknown battery level. For BLE connected observers, or any observer thatis periodically asleep, a “last heard from” indication may be presented(e.g., (approximately) “Now”, “as of 3 m ago,” etc.). A refresh buttonmay also be provided in order to attempt to force an update from aparticular observer. The refresh button may poll the device or pullcurrent device information from a central server. Furthermore, in BLEconnected observers, a distance from appliance/observer may be providedusing BLE ranging. To the extent the mobile app receives new informationfrom a BLE connected device, the mobile application may relay thisinformation to a server so that other mobile phones with permission toaccess the observer state may obtain the most up-to-date state for theobserver via the Internet even thought the observer is only able tobeacon or communicate with short range Bluetooth signals. The mobileapplication may also provide a settings button such as the gear buttonshown in the top left corner of FIG. 12B, and/or a button to add a newobserver device as shown by the + icon in the top right corner of FIG.12B. Pressing the + button presents the add new device view controllerdialog shown in FIG. 12C.

For purposes of illustration and without limitation, with reference toFIG. 12C, an add observer device screen of a Bluetooth Low Energy (BLE)connected observer mobile application is shown. However, the mobileapplication may also be used with other forms of the devices includingWi-Fi, cellular, or other connected observer devices. This viewcontroller searches for unknown devices and allows them to be added tothe main screen. The view controller may also show the physical distanceto the unknown device and optionally sort the device list from closestto furthest or vice versa for example in order to help the user find thedevice that they desire to setup in the case there are multiple unknowndevices that need to be configured.

Once a device is selected to be set up, the user may specify the type ofmachine that the observer will be associated with and observing; and theuser may provide a friendly name for the device (e.g., “HomeDishwasher”, etc.). Optionally, the firmware can be updated on thedevice. In some forms of the invention, the devices are generic devicesand their functionality can be changed simply by choosing the correctfirmware. The firmware can be updated to change the function of thedevice. Firmware for any application can be loaded on to the device. Forexample, if the device is going to monitor a dishwasher one firmware maybe installed. If the device is going to monitor a garage door, otherfirmware may be installed. In other instances, one firmware version mayprovide the functionality for numerous observed devices and a setting inthe observer determines the applicable software settings to use. Forpurposes of further illustration, Source Table 17 provides Objective-CiOS code that corresponds with the application shown in FIGS. 12B and12C. Source Table 1 provides Bluetooth Low Energy connected Arduinocompatible observer device software.

For purposes of illustration and without limitation, with reference toFIG. 12D, an example device detail screen of a Wi-Fi connected observermobile application is shown. Similarly, for purposes of illustration andwithout limitation, with reference to FIG. 12E, another example devicedetail screen of a Wi-Fi connected observer mobile application is shown.However, the mobile application shown in FIGS. 12E and 12D may also beused with other forms of the devices including BLE, cellular, or otherconnected observer devices. At the top of the application screen is anoption to select the observer device in order to view details about thatdevice. There is a drop down menu to select the type of device that isbeing observed. Next, there is a display readout to provide arbitraryinformation such as the name of the device being observed, the stateinformation, and debugging information such as a “run count” (e.g., thenumber of times the main loop has run since resetting the observerdevice, etc.). Below that, there are numerous indicator icons such asthose that correspond to a device state or a low battery warning. Belowthat, there is a graph that can be configured to provide additionaldetailed information about different device parameters including state,type of device, and the measured FFT Average value; the scale of thegraph can be changed with the control below the graph. Below that, thereare sliders to set the FFT Average running threshold value, and a valueto determine how long the “Not Running” determination must persistbefore the state updates to an appropriate state (e.g.,“Running”→“Done”, etc.).

For purposes of illustration and without limitation, with reference toFIG. 12F, a main screen of a Wi-Fi connected observer mobile applicationis shown. However, the mobile application may also be used with otherforms of the devices including BLE, cellular, or other connectedobserver devices. The main screen shows a listing of the observerdevices in tile or row user interface elements along with an indicationof the type of machine or a friendly name that is observed (e.g.,Concorda Dryer, Concorda Washer, New Dryer, Concorda Dishwasher, NewDishwasher, etc.) and the status of each known device (e.g., Done [LowBattery Icon], Done, Load, Load, Load, etc.). Furthermore, a batteryicon can be applied to the information in order to indicate a lowbattery that needs to be replaced or a last known battery level.Touching the user interface element associated with a device willpresent a device detail dialog such as the dialog shown in FIGS. 121 and12J. The mobile application may also provide a settings button such asthe hamburger menu button shown in the top right corner of FIG. 12F.Pressing the hamburger menu button presents the add new device dialogshown in FIG. 12G. The mobile application may also provide a Reportsbutton such as the button shown in the bottom left corner of FIG. 12F.Pressing the Reports button presents the Reports dialog shown in FIG.12H.

For purposes of illustration and without limitation, with reference toFIG. 12G, a device management screen is shown. The screen allows for theobservation, renaming and addition of observer devices. A “lastconnected” or “offline since” time may also be provided for eachobserver device. Observer devices may appear offline during theirfrequent deep sleep cycles for battery power management. Observerdevices may awaken without connecting to the backend server unless astatus update or other server update is required. Pressing the “+ AddNew Device” button will launch a dialog that will prompt the user toconnect to the Wi-Fi hotspot name created by a new observer device insetup mode, and then transfer the household Wi-Fi access pointcredentials (e.g., SSID and password, etc.) to the new observer devicein order to connect the new observer device to join the home Wi-Finetwork and provision the new observer device with the observer backend.Once the new observer device is successfully provisioned, a friendlyname will be requested from the user and the device added to userapplication.

For purposes of illustration and without limitation, with reference toFIG. 12H, a report management screen is shown. A listing of scheduled orrecurring reports is provided with information such as the report name,the scheduling frequency, the date and time of the next scheduledreport, the date and time of the last report that was generated, and anenable/disable selector to enable or disable the particular report.There is also a “+ New Report” button that allows for the generation ofa new report task including the selection of what observer devices, and“virtual pins” or other device parameters are included as well as thefrequency and generation time of the report. The data obtained viareports may be used as training data for the initial training ortransfer learning training of a neural network classifier as discussedelsewhere herein.

For purposes of illustration and without limitation, with reference toFIG. 12I, a device detail settings screen is shown. This screen providesthe hardware version and firmware version reported by the selectedobserver device, as well as the FFT. Average threshold value, RunningRepeat Count to determine how long the FFTA>Threshold before a “Running”determination is made, a Done Repeat Count to determine how long theFFTA<Threshold for a “Done” determination, and an Oven Temp Threshold todetermine, in the case the observer device is configured to observe anoven, if the oven is hot. There is also a data terminal to displayarbitrary debugging information.

For purposes of illustration and without limitation, with reference toFIG. 12J, a device detail display screen is shown. At the top of theapplication screen is a drop-down menu to select the type of device thatis being observed. Next, there are numerous indicator icons such asthose that correspond to a device state or a low battery warning, and anumerical battery voltage readout. Below that, there is a graph that canbe configured to provide additional detailed information about differentdevice parameters including state, the measured FFT Average value, theFFT Standard Deviation value, and the measured Temperature; the scale ofthe graph can be changed with the control below the graph. Below that,there is another graph that can be configured to provide additionaldetailed information about different device parameters including batteryvoltage, and the current orientation of the device (e.g., a 1-6 value,etc.); the scale of the graph can be changed with the control below thegraph. Source Table 18 provides information about the virtual pins thatcorrelate the observer hardware to various parameters stored in theobserver backend that are available to the observer mobile phoneapplication.

Push Notifications to Mobile Devices.

In various forms of the invention, when a state change is detected apush notification may be sent to one or more users associated with theobserver device, or within the proximity of the observer device. Forexample, if a household dishwasher transitions to the “Done” state, apush notification, SMS, or other notification message (e.g., “Thedishwasher is done.”, etc.) may be sent to users associated with thehousehold dishwasher.

In addition, in various forms of the invention, reminder notifications(e.g., “The dishwasher has been done for a while, time to unload it.”,etc.) may be sent if a specific state persists for a period of time. Thenotifications may be sent through a push notification system, or theymay initiated locally on a user-device based on the initial notificationof state change an a lack of further updates about the state of theobserved device.

In addition, in various forms of the invention, a notification may besent to a specific user that was recorded as having last interacted withthe observed machine. For example, a user that loads a loadable machinemay be recognized (e.g., via image recognition, via Bluetooth proximity,via entry of a user ID into the observer device, etc.) and recorded inthe system. When the state changes from “running” to “done,” the systemcan direct the notification message to the recorded user. For purposesof illustration and without limitation, the observer may utilizeBluetooth Low Energy or Wi-Fi to determine who loaded a loadablemachine, and send a notification directed to them when the loadablemachine is ready to unload. Alternatively, the observer may utilizefacial recognition to determine the last user to load a loadablemachine, and send a notification directed to them when the loadablemachine is ready to unload. For purposes of illustration and withoutlimitation, observers may also use BLE or facial recognition todetermine last person near a neglectable machine to send them an alertto turn the machine off if it is in the neglected state. See SourceTables 2 and 3 (“Blynk.notify( . . . )”).

Voice Assistant Notification. In various forms of the invention, when astate change is detected an announcement or notification may be sent toone or more voice assistant devices or remote speaker devices.

For purposes of illustration and without limitation, upon an appliancecompleting cycle, a smart speaker or speaker group (e.g., an Alexa orGoogle Home device, etc.) may provide an announcement (e.g., “thedishwasher is now clean.”, etc.).

For purposes of illustration and without limitation, after an appliancehas completed a cycle, a smart speaker or speaker group (e.g., an Alexaor Google Home device, etc.) may provide an announcement or reminder(e.g., “The washing machine finished a while ago and hasn't beenunloaded. Do not forget to move the close to the dryer!”, etc.).

For purposes of illustration and without limitation, upon an appliancecompleting a cycle but where a dependency is not met (e.g., a washer isdone, the dryer is done but not ready to load because the last loadhasn't been removed, etc.), a smart speaker or speaker group (e.g., anAlexa or Google Home device, etc.) may provide an announcement orreminder (e.g., “Don't forget to unload the dryer. The clothes in thewasher are ready to be dried.”, etc.).

With reference to FIG. 13A, the observer 100 experiences a state changeat act 1302. At act 1304, observer 100 causes its display to update (ifapplicable) and transmits updated state information via a radiointerface (e.g., BLE Beacon/Advertisement, Bluetooth communicationsession, Wi-Fi, cellular, etc.) to an observer backend system accessibleto voice assistant.

For purposes of further illustration and without limitation, upondetermining a state change, an audio file (e.g., way, mp3, etc.) may begenerated with the notification or announcement message. This audio maybe generated on the observer device, on another device, or via an APIover the Internet. The audio file is then uploaded to an internetaccessible URL (e.g., an AWS S3 bucket, etc.). The observer, or anotherdevice, may then initiate a command to one or more smart speakers toplay the audio file. See, e.g., Source Table 19 providing Python sourcecode that causes a Google Home smart speaker to play an audio file.

IV.B Voice Assistant Device Query.

With reference to FIG. 13B, voice assistant receives a voice queryrelating to the observer at act 1312. At act 1314, the voice assistantbackend selects the appropriate voice assistant action to take andqueries the observer backend for relevant observer state or statusinformation. At act 1316, the voice assistant backend receives relevantobserver status information from the observer backend. The voiceassistant backend then formulates a response and transmits the responseto the voice assistant thereby causing voice assistant to annunciate theresponse at act 1318.

For purposes of illustration and without limitation, with reference toFIGS. 13C through 13I, an example voice assistant agent is illustrated.With reference to FIG. 13C, an Intent listing associated with a GoogleDialogFlow for Assistant Agent is shown. An intent “get appliance state”has been created. With reference to FIG. 13D, the “get appliance state”intent details are shown. The parameters of appliance and state aredefined. Training phrases such as “ask mr. wiggle about the dishwasher,”“is the oven hot?,” “is the dishwasher clean,” etc. have been added asexample phrases associated with the intent, and the action andparameters in the training phrases are highlighted. When these phrases,or phrases like these are recognized by the agent, the action andparameter values will be extracted. This information will then be fed tothe response. The Response is defined as a webhook that is enabled forthe event.

With reference to FIG. 13E, the Entities associated with the agent areshown. These are “@appliance” and “@state.” The entity details for“@appliance” are shown in FIG. 13F where appliance names, types andsynonyms are provided. Similarly, the entity details for “@state” areshown in FIG. 13G where state names, types and synonyms are provided.With reference to FIG. 13H, the fulfillment of the response is indicatedto be via webhook (e.g., a URL that provides the response based oninput).

For purposes of illustration and without limitation, with reference toFIG. 13I, an example conversation with the voice assistant agent isprovided. As another example of a voice assistant query, an interactionis dynamically generated based on the state of the observed appliancethat is the subject of the query:

-   -   Q: “[Alexa/Google/etc.], is the dishwasher done?”    -   A (If “Load”): “No, the dishwasher is ready to load.”    -   A (If “Running”): “The dishwasher is still running. I will let        you know when it's done.”    -   A (If “Done”): “Yes, the dishwasher is done and ready to        unload.”

Source Table 20 provides a source code listing for the webhook utilizedwith the voice skill agent illustrated in FIG. 13C through I, above.

In various forms of the invention, to set up a voice assistant skill foruse with their observer devices, a user will initiate a dialog in therelevant voice assistant app (e.g. Amazon Alexa, Google Home, etc.) andselect the relevant Agent or Skill associated with the home observersystem. The user will then provide login credentials for the homeobserver system in order to provide access to the voice assistantsystem. The voice assistant skill will then receive a list of the user'sobserver devices, device names, and/or device ID's. Then, the skill willbe invoked when a relevant query is received. The skill will use theretrieved device IDs, etc., associated with the user in order toformulate an API request to the observer backend to obtain thethen-current state information associated with the machine that is thesubject of the user's query. Upon receiving the information from theobserver backend, the skill can then formulate an audio response that isprovided to the user's smart speaker. Still further, in various forms ofthe invention, the observer backend can initiate notification orannouncement to voice assistant backend for the user when an observerreports a state change.

Location Based Notifications. In various forms of the invention,location-based notifications may be provided. For purposes ofillustration and without limitation, when a user is leaving their house,they may receive a notification from the system if a device is in avulnerable or other state that is not typically appropriate when theuser is leaving the house. For example, if the oven is on and a user isdetected to be leaving the house, then the user may get a notificationthat the oven is on. As another example, when leaving the house, theuser may get a notification that the garage door was left open.

A user's phone can be configured to place a geofence their house. Thephone can then taken an action when it is detected the user is leavingthe geofence area. The phone can contact the observer backend anddetermine if any devices are in a state that is not appropriate.Alternatively, the phone can monitor the visit to home and be notifiedwhen the user is leaving/the visit ended. As another alternative, thephone may monitor its proximity to observer devices known to be presentat the house via BLE ranging or Wi-Fi ranging techniques.

There may be different geofences or geographic settings for differenttypes of devices. For example, the geofence for the oven may be smallerthan the geofence for the garage door. This can give additional time forthe garage door to close. Some devices may rely on geofence rangingwhile others rely on radio-based ranging, etc.

The system may also only alert the last person to leave a house. Forexample, the system can track the members of each household with amobile device.

Member Location State Father Home Mother Home Child Not Home

If the father leaves home while the oven is on, the system may elect notto send an alert because the Mother is also still home. However, thesystem still may elect to send a garage door alert to the father to theextent the garage door is not closed after he leaves. The system mayalso send a garage door alert to a voice assistant at the house to alert(optionally directed specifically to the Mother that is still at home)that the garage door was left open (and that it should be closed). Oncethe door is closed, a follow up alert can be sent to the Father.

Member Location State Father Not Home Mother Home Child Not Home

However, if the mother also leaves the house while the oven is on, thesystem may send a notification to the Mother. If the Mother also forgetsto close the garage door a notification can also be sent.

In systems with actuator or interface to light control system (orgarage, etc.), the observer system can trigger the actuator or turn offlights only when the last person leaves home (or close the garage to theextent there is a garage left open alert). The system may also turn onlights when the first person arrives home.

Interface with Service Availability System. Observer 100 devices may beinterfaced with a service availability system. Home and businesses mayplace observer 100 devices to monitor equipment. When an abnormal stateis detected, the observer 100 can cause a service availability systemassociated with the home or business to reflect that the device isoffline.

For purposes of illustration and without limitation, a Starbuckslocation may install an observer 100 device to observe any of itsvarious machines. If, for example, the nitrogen cold brew system wasinoperative, the observer 100 would detect the issue or an employeecould provide input to the observer to change the state of the machineto Alert/Out of Service. The Starbucks backend server could then reflectthe state of this particular machine and associated product offering atthe particular location in its consumer facing systems. When informationabout the particular location was requested by a potential customer fromthe Starbucks system through for example the Starbucks app or website,the potential customer can be informed that nitrogen cold brew coffee istemporarily not available at the particular location. An alternativenearby location can be suggested by the Starbucks application.

Interface with Procurement System. Observer 100 devices may beinterfaced with procurement systems (e.g., Amazon, WalMart, GroceryStores, etc.). For example, a user purchases consumable items for anappliance from a procurement system (e.g., Amazon.com, etc.). Theparticular detergent or tablets are known to work for a specific numberN runs of the dishwasher (e.g., for a 100 pack of detergent pods N=100,etc.).

The observer 100 device and/or system may count the number of appliancecycles locally and/or on the observer backend. In addition, the observerdevice or observer system or may optionally also report the applianceruns to a procurement system. Alternatively, the observer device orsystem may only contact the procurement system to place an order formore consumables at the appropriate time.

Just in Time Shipping. When the observer 100 reports N (or N—lead_runs)runs have been completed, the procurement system can cause a new orderfor the consumable product to be placed. The value for lead_runs may becalculated based on the shipping time of the product (given a shippingspeed level) and a computed number of average runs per day by the userfor the appliance or the actual number of runs of the appliance duringthat time.

For purposes of illustration and without limitation:

-   -   On Jan. 1, 2019, user purchased 92 dishwasher tablets.    -   The observer 100 detects the dishwasher runs after the tablets        are ordered or delivered (based on tracking history of the        package).    -   On Apr. 1, 2019, 80 runs of the dishwasher have been tallied        since the tablets were delivered. The procurement system infers        that there are 12 tablets left and that the user may be running        low. On average, the user does 5 loads of dishes a week and        there are just over two weeks left before the user will run out.    -   The procurement system suggests that the user purchase        additional tablets by placing a suggestion in the user's        shopping cart, by creating an order and allowing the user time        to cancel the order before it ships, or by including the item in        a pending bulk shipment.

Bulk Shipping. Alternatively, the shipments of consumable products canbe batch shipped monthly. If a consumable product is believed to be inreserve based on reported usage, that product can be excluded from thebatch monthly shipment. For purposes of illustration and withoutlimitation (pods are used for ease of discussion, but volumes of liquidor other measures could similarly be utilized):

-   -   A 10-pack of dishwasher detergent tablets are sent to a user.        After the new tablets arrive, the dishwasher observer device        records 2 runs a week (once every 3.5 days). There is a 10-day        lead time between ordering the tablets and delivery, and the        user is set up on a monthly shipment plan. After about 20 days        (e.g., monthly minus 10-day lead time), it is determined that        the user has used 5 tablets and has 5 tablets remaining. The        next monthly shipment will be allotted a new 10-pack of        dishwasher detergent tablets.    -   A 50-pack of washing machine detergent tablets are sent to a        user. After the new tablets arrive, the washing machine observer        device records 5 runs a week. There is a 10-day lead time        between ordering the tablets and delivery, and the user is set        up on a monthly shipment plan. After about 20 days (e.g.,        monthly minus 10-day lead time), it is determined that the user        has used 10 tablets and has 40 tablets remaining. The next        monthly shipment will not be allotted a new 10-pack of washing        machine detergent tablets because the user still has 40 tablets        remaining and at this rate will have enough to last until the        subsequent shipment at which time the determination to        include/exclude washing machine tablets from the monthly        shipment will be reassessed.    -   A 20-pack of coffee pods are sent to a users with 2-day shipping        time. After the new pods arrive, the coffee machine observer        records each time the machine is run. On average, the machine        records 1 coffee a day for the first week (7 coffees consumed,        13 left), then two coffees a day each subsequent day (9, 11, 13,        15, 17 consumed, 11, 9, 7, 5, 3 left). When the number of pods        remaining minus the product of the consumption rate times the        lead time is less than or equal to a threshold, a signal to        place a replacement order can be initiated. The threshold may be        greater than or equal to 1 or alternatively the consumption rate        times the shipping time. For example, the following table shows        that on at least day 11, a new order for 20 coffees will be        placed (e.g., surplus/deficit threshold is at least 1) so that        on day 12 there will be 3 coffees remaining which will satisfy        the consumption rate for that day, but that a replenishment will        also arrive that day so that there will not be a shortage on day        13.

Surplus/Deficit (accounting for Rate and Coffees/ Rate/ Shipping Day #Day Consumed Remaining Day Time)  1 1  1 19 19  2 1  2 18 −1 16  3 1  317 −1 15  4 1  4 16 −1 14  5 1  5 15 −1 13  6 1  6 14 −1 12  7 1  7 13−1 11  8 2  9 11 −2  7  9 2 11  9 −2  5 10 2 13  7 −2  3 11 2 15  5 −2 1 12 2 17  3 −2 −1

In some forms of the invention, the procurement system may report thenumber of runs from last consumable purchase to observer system, or justthe date or delivery date of the last consumable purchase. This may bebased on a date range or other input. The observer system can thencomplete the calculations and initiate orders based on a run count.

Time Based Reminders. In some forms of the invention, reminders may beset or sent based on the time since the last purchase of a consumableand are commended frequency. For example items certain items (e.g., rocksalt, air filters etc.) typically have a suggested replacement interval(e.g., a month, etc.). These reminders can be set manually in a userinterface of the application, added automatically based on an orderhistory and recommended interval from a procurement system, and form thebasis for reminder alert notifications or voice assistant reminders tochange these or similar items.

API to Query Projects or State Machines. In some forms of the invention,an Application Programming Interface (API) to retrieve and/or locate,query, set, or otherwise manage state machines is provided. For example,some features that may be available via the API might include:

-   -   Get current state from a state machine (e.g., by state machine        ID, etc.)    -   Set current state on a state machine    -   Get history of states from a state machine    -   Get/subscribe to notifications of a state transition from a        state machine    -   Get state machine ID by location (e.g., lat/lon, zip code, etc.)        and machine type (e.g., dishwasher, parking meter, etc.).    -   Get state machine ID by location and image of machine (e.g.,        using image recognition)    -   . . . etc.

Interaction with Third-Party APIs. The API may be used to provide aninterface to systems like “If This Then That” (IFTTT) that bridge theobserver backend API with APIs provided by other services, or theobserver backend may interface directly with other APIs. This allows fortriggers to execute on state change or arrival/departure/dwell on state.For purposes of illustration and without limitation, some exampletriggers are:

-   -   If OBSERVER DEVICE transitions to state SELECT STATE, then        ACTION PROVIDED BY ANOTHER API.    -   If OBSERVER DEVICE transitions from state SELECT STATE, then        ACTION PROVIDED BY ANOTHER API.    -   If OBSERVER DEVICE is in state SELECT STATE for at least time        TIME PERIOD, then ACTION PROVIDED BY ANOTHER API.        For purposes of illustration and without limitation, if the        dishwasher observer transitions to state done, a Phillips HUE        light may be configured to change its color to green        momentarily, or persistently until the dishwasher is again ready        to load. As another example, if the garage door observer        transitions to an open state, a smart speaker device can be        instructed to play a sound. If the garage door observer        transitions from the open state or to the closes state, the        sound can be instructed to stop. As yet another example, If the        oven observer device is in the hot state for more than 6 hours,        then an SMS message can be configured to be sent to the home        owner.

V. State Machine and Neural Network Model Training and Editing Tools.

In various forms of the invention, a graphical editing tool may be usedto create a project, add state machine(s) to the project, and mapclassifier(s) to various parts of the state machine(s). The tool mayalso be used to deploy, manage, debug, and retrain observers andclassifiers.

FIG. 14 depicts an exemplary flowchart for the creation, training,running, and refining of a machine learning or AI state machine model.

FIG. 15A depicts an exemplary Graphic User Interface (GUI) for creatinga new state-machine model project for an observer device using AIclassifiers.

FIG. 15B depicts an exemplary Graphic User Interface (GUI) for editing,training, running, and refining a state-machine model for an observerdevice using AI classifiers.

V.1 Create a New Project

V.1A State Machine Selection.

After a project is created, a state machine model is created.

Manual State Machine Creation. A user may draw a state machine orcomplex state machine from scratch using a pallet of tools including:states, transitions, arrows, memory states, etc.

State Machine Library. A user may select a state machine from a libraryof pre-defined templates or prototypes to insert into the projectworking area. These state machines may be generic types such as thosedescribed above (e.g., two-state, three-state, neglectable machine,cyclic machine, abnormal state detector, etc.), or they may bemore-specific pre-defined machines associated with common types ofmachines (e.g., dishwasher, washer, dryer, refrigerator, oven, garage,etc.).

-   -   Table of known objects and prototype state machines    -   Look up objects to get relevant state machine prototypes    -   Prototypes may be simple state machines that need to have        classifiers added, or may have suggested classifiers already        attached to at least one state or transition    -   Automatically associate state machine models with known objects        (e.g., a specific model of machine associated with a specific        state machine, etc.).    -   Using image recognition to assign a state machine to a        recognized object type    -   Image recognition determines machine    -   Looks up associated state machine    -   Assigns default state, or determines “initial” state based on        observed activity    -   Reports location, machine, state to remote server where the        information is stored in a database and may be used by other        devices to identify the particular machine.    -   Automatic Identification of State Machine for a Specific Machine    -   Lookup based on location, image, etc.    -   May get last reported state as an initial state    -   Run classifiers to determine if still accurate.

V.1.B Specify States

-   -   Pre-Loaded from Library    -   Add States with Tool Pallet/Menus/Drawing    -   Delete States State Details    -   Attached Classifiers.

V.1.C Specify Transitions

-   -   Pre-Loaded from Library    -   Add Transitions with Tool Pallet/Menus/Drawing    -   Delete Transitions    -   Transition Details    -   Attached Classifiers.

V.1.D Assign Classifiers and Sensors

-   -   Pre-Loaded from Library    -   Add Classifier with Tool Pallet/Menus/Drawing        -   Selection of Classifier            -   From classifier library            -   Filter based on sensor type available            -   Filter based on observer device type (e.g., available                edge compute power, etc.)            -   Selection of sensor data to provide to classifier.                -   Pre-filtered based on sensor type available.                -   Select options for sensor data.                -   Frequency                -   Filters on the data                -    Video bounds for the classifier (e.g., only portion                    of captured image).                -    Audio filters for audio data, etc.    -   Delete Classifier    -   Classifier Details

Define Transitions/Classification Using NLP. In some forms of theinvention, the states and transitions have may classifiers attached tothem using natural language processing. The classifiers may be definedbased on a user editing a dialog, and selecting any or a combination ofthe features to input into the classifier, any preprocessing that needsto be done to the input, and a standard classifier (e.g., ImageNet,UCF-101, etc.), a custom classifier, a combination of classifiers (e.g.,ImageNet, and UCF-101, etc.), or any other information.

The classifier may also be selected automatically based on the type ofinput expected and a sentence describing the action that will define thestate or trigger the transition.

-   -   “a person loads a dishwasher”    -   “the dishwasher is running”    -   “the dishwasher stops running”    -   “a person unloads a dishwasher”    -   “a doctor or nurse enters the room”    -   “a doctor or nurse exits the room”

Using Natural Language Processing (NLP), a lookup table, or any othermethod, the appropriate classifier or combination of classifiers can beselected in order to define the state or transition.

For purposes of illustration and without limitation, a classifier thatis based off UCF-101 using transfer learning may be generated based oninformation that contains “loading”, “running”, “unloading”, “enteringroom”, “exiting room”, “appliance turned on” classifications. A user mayonly need to specify in the transition arrow in the GUI that the“appliance turned on” and the application will find the best availableclassifier to use as a basis. This classifier may be improved withtransfer learning.

V.2 (Re-)Training Data Collection

-   -   For each classifier as needed.    -   Collect data for each possible classification.    -   For each state as needed.    -   Collect data for each possible state.

V.3 Training or Transfer Learning

Assign (Re-)Training Data to States/Transitions. If not already done aspart of 2, assign the (re-)training data to the classifiers associatedwith transitions and states in the GUI.

Initiate Training/Transfer Learning. For each classifier, training or(re-training such as transfer learning) will be initiated.

The classifiers can be trained independently from one another.

May be done on device or may be done remotely on a server.

V.4 Deploying and Inference (with or without the GUI Application)

Deploying The State Machine and Classifiers to Observer Devices.

The state machines and classifiers can be deployed together orseparately and may be deployed to the same or different portions of theobserver system network.

State Machine on Cloud/Remote State Machine on State Machine on ServerLocal Server Edge Device Classifier on The Classifier and The Classifiercan The Classifier can Cloud/Remote Server State Machine may be deployedto a be deployed to a both be deployed to Remote Server Remote Server aRemote Server. while the State while the State Machine is hosted Machineis hosted on a Local Server. on an Edge Device. Classifier on TheClassifier can The Classifier and The Classifier can Local Server bedeployed to a State Machine may be deployed to a Local Server while bothbe deployed to Local Server while the State Machine the Local Server.the State Machine is hosted on a is hosted on an Remote Server. EdgeDevice. Classifier on The Classifier can The Classifier can TheClassifier Edge Device be deployed to an be deployed to an and StateEdge Device while Edge Device while Machine may both the State Machinethe State Machine be deployed to the is hosted on a is hosted on a EdgeDevice. Remote Server. Remote Server.

Deploying Classifiers.

-   -   May deploy to cloud        -   Observers merely send sensor data to server where it is            classified.    -   May deploy to local server (e.g., a home router, hub, raspberry        pi, mobile device, phone, etc.) on which the classifiers are        run.    -   May deploy to edge devices (i.e., the observer device, mobile        device, phone, etc.).

Deploying State Machines.

-   -   Cloud    -   Local Server    -   Edge Device/Observer Device        -   Getting state machine onto observer:    -   Take state machine, converted to code that is compiled and then        loaded onto observer device; or    -   Send state machine definition to observer device; or    -   Other methods        -   Deploy to cloud system server        -   Push to fleet of devices        -   Push to specific device        -   Directly load onto device

b. Model Runs/Feed Forward

-   -   Model runs        -   For each classifier collect a history of inputs and            associated determinations, resulting states and transitions            for later auditing, flagging/tagging, and re-training.

Observer Memory Management. Switching out classifiers in memory based onstate (this may be used in any deployment, not just those through theGUI). State machine on observer device may cause a classifier(s) to beloaded into a neural network processor, or another processor on theobserver device.

-   -   When on a first state, only the classifiers that are relevant to        that state, or transitioning from that state may be evaluated on        the device.    -   Alternatively, more classifiers or all classifiers may be        evaluated at any time in order to ensure that the state machine        is in the proper state and is not “lost.”

Publish/Broadcast State

The instructions for where the state is published to (what backends,what protocols) may be edited so that the relevant systems can benotified on state change.

Instructions relating to publishing other identifying information aboutthe observed object so that other third parties may look up the statewhen they encounter the object in the real world. Passwordprotect/privacy so that only authorized people can examine theinformation relating to the observed device

V.5 Audit and Refine

Input History (e.g., Images, Sound Clips, Video, Video (MotionDetection)

History saved with whether or not it triggered the transition

User can audit/review the input and the result,

If there are false positives or negatives for any transition, then thetags can be updated and the classifier re-trained or refined based onthe user-feedback

Feed re-tagged data back into classifier training portion

Re-train/transfer learning

a. Review Input/Classification History

b. Make Corrections/Re-Tag Data

V.6 GUI Use Case Example: Crop Observer View to Focus on Specific Areaof Image or Sensor Input

In the case of an observer that is observing multiple machines, theobserver input can be cropped, masked, or otherwise filtered in order todefine multiple observed items.

For purposes of illustration and without limitation, a video observerpositioned in a hospital hallway may be watching several patient roomdoors and monitoring several features with each room:

-   -   Is the room occupied by a patient?    -   Is the room ready for a patient?    -   Has a doctor or nurse checked on the patient recently?    -   Is a patient being visited by guests?

In this example, the observer has at least a video feed, that has atleast one (but possibly multiple) patient room doors. For each patientroom watched by the observer, the hallway and door for the room may bemasked and associated with a project, a state machine, or a group ofstate machines. The project may be given a name or identifier (e.g.,“Patient Room 102”) so that it can be remotely located and monitored byhospital staff, a quality assurance department, an ApplicationProgramming Interface, etc.

A state machine may be associated with the project to answer thequestion “is the room occupied by a patient?”. This may create, forexample, a “neglectable machine” or a “single user machine.” A customstate machine may also be defined. In the case of a “single usermachine,” an “in use” transition defined by whether a “patient” has beendetected “entering a room.” Similarly, the “Available” transition may bedefined by whether the “patient” has been detected “leaving a room.

The transitions may be determined based on a custom AI classifier, or acombination of custom or semi-custom transfer learning trainedclassifiers such as ImageNet (or a modified ImageNet or similar network)and UCF-101 (or a modified UCF-101 or similar network).

There may be several different “Alert” transitions or additional statemachines assigned to the room. (Second observer piggy back on to thesame state machine/project as an additional input. A camera in the room.Alert if the patient is flailing arms/in distress/machine beeping/etc.)

Wrap Entire State Machine in Conditional Statement

Enable State Machine Only if Another State Machines State is (not) aSpecific State or Set of States.

At least one “Alert” transition of the “single user machine” may bedefined by the output of an “abnormal state detector” state machine, orthe alert may be part of another state machine, such as a separate“generic machine” or “abnormal state detector” state machine.

The abnormal state detector may have its normal state be defined aswhether a “doctor or nurse” has “entered the room” “within the lasthour.” Again, this can be manually specified with a custom classifier,or a combination of pre-trained or transfer-learning trained modifiedclassifiers. The time component may be defined by a timer and anobservation history of the observer or project. To the extent that thecondition is not determined to have happened by the classifier, thestate machine will transition to its alert state. Alerts may be sent tohospital staff and the family of the patient, for example. While in thealert state, once a “doctor or nurse” “enters the room”, the statemachine may transition back to its normal state.

A state machine may be associated with the project to answer thequestion “is the room ready for a patient?”

V.7 State Machine Store or Exchange

Good state machines for various use cases can be exchanged or sold

Move to Situational/Location Based area: Can be placed on mobile robotthat roams building and checks on appliances and updates the statediagram for each appliance.

VI. Use Cases

Countless types of observer hardware devices may be implemented invarious environments, to observe any imaginable object. The observedobjects can be modeled using state machines that are driven withappropriate classifiers created as described herein. Some of thepossible applications in different environments are illustrated below.

VI.1 Home Applications

VI.1.A Washer/Dryer Example:

-   -   Camera in utility room aimed at washer and Dryer.    -   Recognizes the state of each machine (state machine diagram)    -   Uses AI image recognition (LSTM with Convolutional neural        network) to determine state of each machine by looking at people        interacting with the machines and the indications on the control        panels of each machine.        -   May have database of common machines that it recognizes/has            been trained on.    -   Sends notification to users when the machine state changes/if        there is an error/upon request:        -   “Washing machine is off balance”        -   “Washing machine is ready to unload”        -   “Washing machine is empty”        -   “Washing machine is currently running. Estimated completion            in 5 minutes.”    -   Can recognize the user that interacted with the machine last,        and direct notifications to that user.        -   Recognized James loaded the washing machine, and it has            become off balance due to an “OFB” indication on the control            panel. Notification sent to James' phone to check washer            balance and restart.    -   Similar examples for Dryer.    -   Also can detect unknown errors or other issues:        -   Unexpected water on floor.        -   Unknown issue—for example if something fell over in the            utility room and it was not something the AI was used to            detecting.

VI.1.B Saw Example:

-   -   Computer coupled to camera which is aimed at an area of interest        (e.g., saw blade).    -   Convolutional neural network (or other type of computer        algorithm) trained (or configured) to recognize a dangerous        situation (e.g., hand too close to saw blade, etc.)    -   Configured to activate the actuator in the danger condition to        reduce risk of accident (eg open relay to take power away from        saw)

VI.1.C Pool Pump Machine/Legacy Meter Example

An image-based observer may be trained on a legacy meter or other devicewith dials or other analog or digital readouts. The observer can betrained to correlate the display of the legacy device with states andtransitions to provide an alert to a user if the machine needsattention. For example, with an observed pool pump, an abnormal statedetector may be associated with a state transition from normal toabnormal when the observed pool pump pressure needle is outside ofnormal ranges.

VI.1.D Home Chore/Task Assignment

System can delegate tasks to various household members on a random,sequential, assigned or any other basis.

For example, a list of household members may be kept in the system. Fora dishwasher that is ready to load, a household member can be assignedto always load the dishwasher, or one can be assigned randomly or in asequence so that everyone shares the responsibility. The same can bedone for unloading the dishwasher.

The setting for each appliance can be set and the information stored inthe system.

When the observer transitions to a state that calls for userintervention of a specific appliance, a user is determined and thespecific user may be notified that it is their responsibility to performthe action. For example, a dishwasher cycle completes, the systemdetermines it is a husband's turn to unload the dishwasher. A pushnotification is sent to the husbands phone and/or a voice assistantmessage or notification is posted to the husband's voice assistantaccount.

Industrial/retail/quick service restaurant applications.

VI.1.E Television Emotion Tracking

Intelligent observer can be mounted on television to obtain useremotional feedback about the shows that are being watched.

Can identify the television show being watched based on sub audibletones in the show or other sound recognition.

Can identify the emotion on observers faces and correlate with variousaspects of the show to determine engagement

VI.1.F Advertising/Purchase on Device

In some forms of the invention, advertisements may be placed on theobserver screen associated with appliance. These may be directed to anew product for the user (with optional discount for buying it now), ora product that the user has used in the past or has indicated is apreferred product. A user may make a purchase by pressing a button orotherwise indicating that they would like to purchase the advertiseditem through a voice assistant or application interface.

For example:

-   -   favorite dishwasher detergent advertisement, or advertisement        for a new detergent, may be displayed on the device. Pressing a        button may cause the item to be purchased or added to a shopping        cart.    -   For washer, could be clothes pods or detergent, fabric softener,        etc.    -   For dryer, could be dryer sheets to prevent static electricity.    -   For oven, may be cooking tins or other consumable or cookable        products.    -   If an advertisement for a new product is provided, it may also        provide an option to purchase established product instead.

May also be free samples, etc:

-   -   Press Button to Try    -   New ______ Dishwasher Tabs!    -   Rated Five Stars on Amazon.

VI.1.G Advertising Based on Machine State

Machines associated with an account

-   -   Ads can be provided to a user based on the observed states or a        history of observed states associated with machines that the        user is associated with or is known to manage or own    -   Ad for dishwasher detergent based on state (or state history of        dishwasher)    -   Ad for paper towels based on state (or state history) of an        observed paper towel holder        -   Based on an observed brand of paper towel wrapper        -   Based on a purchase history of paper towels associated with            the user    -   Nearby machines        -   Ads can be provided based on the observed state of nearby            machines    -   Advertising may be provided to any device associated with the        user        -   Computer        -   Phone        -   TV        -   Observer devices with display        -   Watch/wearable computers

VI.1.H PIR Sensor

In some forms of the observer devices, a PIR sensor may be incorporatedso that it can sense when a human is nearby. The PIR sensor can signalan interrupt that wakes the observer. The observer can cause a pushnotification or announcement when human nearby as to status change forthe observed appliance (e.g., as a human approaches a washing machinethe human can be alerted audibly of the washing machine state if therehas been a state change e.g., “this washing machine is now clean”,etc.). When everyone is out of the house, each observer can also becomea motion/burglar detector (e.g., abnormal state machine, etc.).

VI.1.I Trick or Treat

Example: Create state machine for new trick or treater and to operate anactuator that deploys a serving of candy per trick-or-treater.

VI.2 Virtual Reality/Augmented Reality/Mixed Reality Applications

VI.2.A AI Observer—Virtual Input Devices

-   -   Use the exterior cameras to do image recognition on feet to        determine input.    -   A virtual object state machines is created based on real world        input as detected by observer        -   For example, a virtual set of rudder pedals (virtual) can be            created and associated with a state machine, with foot            (real) determining input to the virtual object state            machine.    -   Can be based on for example the inside out tracking cameras of        the oculus quest, or an external camera or depth sensor like a        Kinect device.

VI.2.B 3D Telepresence

-   -   Device or set of devices that is put in a room (eg conference        room) that creates a 3D video model of the participants    -   The 3D video model can be streamed or otherwise sent to a remote        device over a network such as an oculus    -   The 3D video model can be generated from multiple photo angles        using techniques such as.    -   Use image recognition techniques to highlight person talking        etc. and annotate participants.

VI.2.C Remote Telepresence Devices

-   -   Telepresence devices, observers, placed in the world in various        locations (e.g., like scooter share scooters, etc.)    -   May be control them from afar with smartphone app to see what is        happening somewhere etc.    -   for a conference or meeting that cannot get to can send a drone        -   get a drone near a friends house and have it fly to you with            small item from friend        -   power pads that the drones land on and are based from            -   people sign up to host a power pad    -   Deploy a state machine to a remote observer device for a fee to        rent the time on the observer.    -   Rent a video feed from an observer device to run an observer        state machine model remotely (e.g., rent feed from a red light        camera to monitor an intersection for an event, etc.)

VI.2.D Capacitive Keyboard

-   -   Problem—Hard to Type in VR because cannot see keyboard    -   Solution—Keyboard that senses hand position independently of key        presses so that a virtual display of hands can be placed on        display in VR    -   Keyboard with:        -   Microcontroller        -   Wireless or wired interface to computer/VR headset,            preferably wireless (Bluetooth or ISM)        -   Battery/power distribution circuitry        -   capacitive keys so that finger proximity can be sensed            independently of key press            -   Inverse Kinematics for determining how to display hand                on keyboard        -   Keys depress to activate as expected to type a character        -   Circuitry to locate device in space relative to headset,            -   6DOF/9DOF sensors, and/or            -   IR Light emitters to assist with tracking of the                position and orientation of the keyboard

VI.3 Automotive Applications

-   -   Observer trained to recognize dangerous behaviors (texting while        driving)        -   Leaving infant or pet in a car        -   Sensor in car seat    -   Camera/spherical camera etc. in car    -   State machine to track baby placed in seat, removed from seat        -   Rear facing car seats may shield baby from camera view        -   Tracking the baby being placed in the car seat and then            being removed (or not) parent can be alerted if the car is            stopped and shut down without baby being removed from the            car seat        -   Detect baby crying sound-> Alert.

VI.4 Traffic Applications

VI.4.1 AI Traffic Light

Light interfaced with cameras or lidar etc. Determines where pedestriansare relative to the intersection. Sensors may be placed anywhere on thestoplight system. One sensor or more may be utilized to obtain fullsensor coverage of the environment.

-   -   Sense Environment    -   Determine        -   Pedestrian Locations        -   Automobile Locations    -   Anticipate        -   Pedestrian Movement        -   Traffic Movement    -   Actuate/Modify        -   Modify the normal operation of the light.        -   Actuate the signals to optimize safety and traffic flow.    -   Enforce        -   If pedestrians try to take advantage of the system to cut            the line/block traffic, then their picture can be recorded,            and facial recognition used to issue tickets/citations.        -   If cars cause an issue, pictures of the car, driver and            license plate can be recorded, and tickets/citations issued            if necessary.

The system can orchestrate the lights accordingly to ensure that cars donot inadvertently conflict with pedestrian traffic.

For example, bike waiting at an intersection to cross street.

Light would normally signal for cars to go straight or turn left.Bike/pedestrian may also be told to walk at that point as well. A carturning left would potentially conflict with the bike/pedestrian.

With the AI light system, the system would recognize that a car shouldnot turn left until the bike/pedestrian is at least halfway across thestreet, and will ensure that the light for left turn remains red untilthere is no conflict with the pedestrian traffic.

VI.4.2 Wrong Way Driver Prevention

-   -   Detect someone driving the wrong way onto a freeway onramp        (abnormal state detector).    -   Deploy tire shredders.

VI.4.3 Smart Gated Community Gate

-   -   Recognizes authorized cars and opens gate, records access,        notifies appropriate individuals.    -   Recognizes service vehicles: checks schedule, if authorized        opens gate, records access, notifies appropriate individuals; if        not authorized, notifies appropriate individuals to request        access

VI.5 Industrial Applications

-   -   Monitor factory machines.    -   Monitor factory workers.    -   Monitor batch processes.

VI.6 Retail Applications

In various forms of the invention, observer devices may be used inretail establishments.

VI.6.A Facial Recognition for Entry to Places During Rush Hour

-   -   Facial Recognition for Entry to Places During Rush Hour    -   Popular places/coffee/food/store.    -   Charge for premium times to control crowd.    -   Card, app on mobile device, facial recognition controls entry or        ability to purchase items during those times.

VI.6.B Facial Recognition for Customer Relation Management

Intelligent observer(s) with a camera is placed in a commercial setting(e.g., store, café, department store, government office). It is pairedwith an application on a mobile device (optional) with location trackingreports location to a central server (e.g., Facebook) that also hasfacial image data for the user (e.g., Facebook). The intelligentobserver utilizes facial recognition on the images it is receiving fromthe camera to identify the customers or users in the store. Intelligentobserver can determine the locations in the store that the user isinterested in (e.g., in the case of a department store). Retail locationcan then:

-   -   Offer prioritized service    -   Pull up the customer file    -   Push notifications to the identified customer's phone    -   Collect data about the locations the user goes to in the store,        and the time spent in those locations.

VI.6.C Facial Recognition for Emotion/Survey of Employee Effectiveness:

-   -   Device to correlate facial expressions to level of satisfaction    -   At point of sale terminal—correlated to server or employee;    -   Mobile device associated with user—location tracked to        correspond to establishment

VI.6.D Queue Management. Video recognition of people with textualdescription of person coupled with instructions: “Man in black shirtwith glasses and beard, please proceed to register 15.”

VI.6.E Rating button box that uses video camera and facial recognitioninstead of buttons. Observer is trained to observe people and obtain avideo thumbs up/thumbs down rating or a sentiment value based on facialexpression. Intelligent observer device placed outside of business orroom with sign to rate their experience with thumbs up/down orsmile/frown. Facial and hand expression recognition is used to determinemood of people exiting or response to sign as a positive or negativeresponse. If someone speaks directly at the box, then voice recognitioncan be used to transcribe text spoken directly to the box as a comment.Data regarding the response is made available via the internet so thatquality assurance people can be dispatched as needed.

VI.6.F Room occupied detector for lighting control. Motion sensor forlighting control. Keeps count of people in the room. If 0 people, lightsoff

Conclusion. As those skilled in the art will appreciate, many aspects ofthe invention, and the various forms of the invention, can beneficiallybe practiced alone and need not be coupled together. Unless specificallystated otherwise, no aspect of the invention should be construed asrequiring combination with another aspect of the invention in practice.However, those skilled in the art will also appreciate that the aspectsof the invention may be combined in any way imaginable to yield one ofthe various forms of this invention.

I claim:
 1. A device comprising: a) a housing including a magnetic oradhesive portion adapted to attach the device to an observed machine; b)a processor; c) a triple-axis accelerometer; and, d) a wirelesscommunication interface; wherein the processor is configured to: i)periodically record a series of measurements from at least one axis ofthe triple-axis accelerometer over a period of time, ii) process thosemeasurements using a Fourier transform, iii) compute the average valueof the Fourier transform result, iv) compare the average value to athreshold value, v) determine, based on the comparison, whether thedevice should cause a transition in a state machine model of theobserved machine, and vi) if the determination is yes, cause the statemachine transition to occur.
 2. A device comprising: a) a processor; b)an imaging sensor; c) a microphone; and, d) a wireless communicationinterface; wherein the processor is configured to: i) periodicallyrecord an image from the imaging sensor, ii) periodically record a soundfrom the microphone; iii) convert the sound into a spectrogram; iv) passthe image and sound through a deep neural network to determine theprobability that an observed machine has changed state, v) determine,based on the result, whether the device should cause a transition in astate machine model associated with the observed machine, and vi) if thedetermination is yes, cause the state machine transition to occur.